Context aware surgical systems

ABSTRACT

Systems and methods are provided in which devices that are employed during a medical procedure are adaptively configured during the medical procedure, based on input or feedback that is associated with the current state, phase or context of the medical procedure. In some example embodiments, the input is obtained via the identification of one or more medical instruments present within a region of interest, and this input may be employed to determine configuration parameters for configuring the device. In other example embodiments, the input may be based on the image-based detection of a measure associated with the phase or context of the medical procedure, and this input may be employed to adaptively control the device based on the inferred context or phase of the medical procedure. In other embodiments, images from one imaging modality may be employed to adaptively switch to another imaging modality.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.61/801,530, titled “SYSTEMS, DEVICES AND METHODS FOR PLANNING, IMAGING,AND GUIDANCE OF MINIMALLY INVASIVE SURGICAL PROCEDURES” and filed onMar. 15, 2013, the entire contents of which is incorporated herein byreference, and to U.S. Provisional Application No. 61/800,695, titled“EXTERNAL VIDEO SCOPE FOR PORT-BASED SURGICAL PROCEDURES” and filed onMar. 15, 2013, the entire contents of which is incorporated herein byreference and to U.S. Provisional Application No. 61/800,787, titled“POLARIZED LIGHT IMAGING DEVICE” and filed on Mar. 15, 2013, the entirecontents of which is incorporated herein by reference, and to U.S.Provisional Application No. 61/800,911, titled “HYPERSPECTRAL IMAGINGDEVICE” and filed on Mar. 15, 2013, the entire contents of which isincorporated herein by reference and to U.S. Provisional Application No.61/801,746, titled “INSERT IMAGING DEVICE” and filed on Mar. 15, 2013,the entire contents of which is incorporated herein by reference and toU.S. Provisional Application No. 61/801,143, titled “INSERTABLE MAGNETICRESONANCE IMAGING COIL PROBE FOR MINIMALLY INVASIVE CORRIDOR-BASEDPROCEDURES” and filed on Mar. 15, 2013, the entire contents of which isincorporated herein by reference and to U.S. Provisional Application No.61/801,282, titled “SYSTEMS AND METHODS FOR PATHOLOGY TRACKING” andfiled on Mar. 15, 2013, the entire contents of which is incorporatedherein by reference and to U.S. Provisional Application No. 61/800,155,titled “PLANNING, NAVIGATION AND SIMULATION SYSTEMS AND METHODS FORMINIMALLY INVASIVE THERAPY” and filed on Mar. 15, 2013, the entirecontents of which is incorporated herein by reference to U.S.Provisional Application No. 61/818,255, titled “INSERT IMAGING DEVICE”and filed on May 1, 2013, the entire contents of which is incorporatedherein by reference and to U.S. Provisional Application No. 61/818,325,titled “INSERTABLE MAGNETIC RESONANCE IMAGING COIL PROBE FOR MINIMALLYINVASIVE CORRIDOR-BASED PROCEDURES” and filed on May 1, 2013, the entirecontents of which is incorporated herein by reference, and to U.S.Provisional Application No. 61/818,280, titled “SYSTEMS, DEVICES ANDMETHODS FOR PLANNING, IMAGING, AND GUIDANCE OF MINIMALLY INVASIVESURGICAL PROCEDURES” and filed on May 1, 2013, the entire contents ofwhich is incorporated herein by reference, and to U.S. ProvisionalApplication No. 61/818,223, titled “IMAGING ASSEMBLY FOR ACCESSPORT-BASED MEDICAL PROCEDURES” and filed on May 1, 2013, the entirecontents of which is incorporated herein by reference, and to U.S.Provisional Application No. 61/924,993, titled “PLANNING, NAVIGATION ANDSIMULATION SYSTEMS AND METHODS FOR MINIMALLY INVASIVE THERAPY” and filedon Jan. 8, 2014, the entire contents of which is incorporated herein byreference

BACKGROUND

The present disclosure is generally related to image guided medicalprocedures.

In the field of surgery, imaging and imaging guidance is becoming a moresignificant component of clinical care, from diagnosis of disease,monitoring of the disease, planning of the surgical approach, guidanceduring the procedure and follow-up after the procedure is complete, oras part of a multi-faceted treatment approach.

Integration of imaging data in the surgical suite has becomecommon-place for neurosurgery, where typically brain tumors are excisedthrough an open craniotomy approach guided by imaging. The data that isused typically consists of CT scans with associated contrast (iodinatedcontrast), and MRI scans with associated contrast (gadolinium contrast).Systems provide a means to register the imaging data sets together, andregistration methods to translate the three dimensional imaging space tothe three dimensional space of the patient and tracking of instrumentsrelative to the patient and the associate imaging data by way of anexternal hardware system such as a mechanical arm, or a radio-frequencyor optical tracking device.

SUMMARY

Systems and methods are provided in which devices that are employedduring a medical procedure are adaptively configured during the medicalprocedure, based on input or feedback that is associated with thecurrent state, phase or context of the medical procedure. In someexample embodiments, the input is obtained via the identification of oneor more medical instruments present within a region of interest, andthis input may be employed to determine configuration parameters forconfiguring the device. In other example embodiments, the input may bebased on the image-based detection of a measure associated with thephase or context of the medical procedure, and this input may beemployed to adaptively control the device based on the inferred contextor phase of the medical procedure. In other embodiments, images from oneimaging modality may be employed to adaptively switch to another imagingmodality.

Accordingly, in one aspect, there is provided a computer implementedmethod of adaptively and intraoperatively configuring a device usedduring a medical procedure, the method comprising:

-   -   identifying a medical instrument during the medical procedure;    -   obtaining one or more customized configuration parameters for        adaptively configuring the device during the medical procedure,        where the customized configuration parameters are selected based        on the identity of the medical instrument; and    -   configuring the device according to the customized configuration        parameters.

In another aspect, there is provided a system for adaptively andintraoperatively configuring a device used during a medical procedure,comprising:

-   -   a data storage device comprising customized configuration        parameters for adaptively configuring one or more devices during        the medical procedure;    -   a control and processing system interfaced with the device and        the data storage device, said control and processing system        comprising one or more processors and memory coupled to said one        or more processors, said memory storing instructions, which,        when executed by said one or more processors, causes said one or        more processors to perform operations comprising:    -   identifying a medical instrument during the medical procedure;    -   obtaining, from the data storage device, one or more customized        configuration parameters for adaptively configuring the device        during the medical procedure, where the customized configuration        parameters are customized based on the identity of the medical        instrument; and    -   configuring the device according to the customized configuration        parameters.

In another aspect, there is provided a computer implemented method ofadaptively configuring a device used during a medical procedure, themethod comprising:

-   -   obtaining one or more images of a region of interest associated        with the medical procedure;    -   processing the one or more images to identify a context measure        associated with the current state of the medical procedure;    -   obtaining one or more customized configuration parameters for        adaptively configuring the device during the medical procedure,        where the customized configuration parameters are customized        based on the context measure; and    -   configuring the device according to the customized configuration        parameters.

In another aspect, there is provided a system for adaptively andintraoperatively configuring a device used during a medical procedure,comprising:

-   -   a data storage device comprising customized configuration        parameters for adaptively configuring one or more devices during        the medical procedure;    -   a control and processing system interfaced with the device and        the data storage device, said control and processing system        comprising one or more processors and memory coupled to said one        or more processors, said memory storing instructions, which,        when executed by said one or more processors, causes said one or        more processors to perform operations comprising:    -   obtaining one or more images of a region of interest associated        with the medical procedure;    -   processing the one or more images to identify a context measure        associated with the current state of the medical procedure;    -   obtaining one or more customized configuration parameters for        adaptively configuring the device during the medical procedure,        where the customized configuration parameters are customized        based on the context measure; and    -   configuring the device according to the customized configuration        parameters.

In another aspect, there is provided a computer implemented method ofadaptively controlling a first imaging modality and a second imagingmodality during a medical procedure, the method comprising:

-   -   while obtaining first images with the first imaging modality,        intermittently obtaining one or more second images with the        second imaging modality;    -   processing the second images to calculate, for a plurality of        regions within the second images, an image measure associated        with the second imaging modality; and    -   in the event that the image measure for one or more regions is        within a pre-selected range, increasing the rate of acquisition        of the second images.

In another aspect, there is provided a system for adaptively controllingone or more imaging devices during a medical procedure, comprising:

-   -   a control and processing system interfaced with the one or more        imaging devices, said control and processing system comprising        one or more processors and memory coupled to said one or more        processors, said memory storing instructions, which, when        executed by said one or more processors, causes said one or more        processors to perform operations comprising:    -   obtaining first images with a first imaging modality and        intermittently obtaining one or more second images with a second        imaging modality;    -   processing the second images to calculate, for a plurality of        regions within the second images, an image measure associated        with the second imaging modality; and    -   in the event that the image measure for one or more regions is        within a pre-selected range, increasing the rate of acquisition        of the second images.

In another aspect, there is provided a computer implemented method ofadaptively controlling one or more imaging devices during a medicalprocedure, the method comprising:

-   -   a) obtaining one or more first images with a first imaging        modality;    -   b) processing the first images to calculate, for a plurality of        regions within the first images, an image measure associated        with the suitability of a second imaging modality;    -   c) in the event that the image measure for one or more regions        lies within a pre-selected range, acquiring one or more second        images with the second imaging modality.

In another aspect, there is provided a system for adaptively controllinga one or more imaging devices during a medical procedure, comprising:

-   -   a control and processing system interfaced with the one or more        imaging devices, said control and processing system comprising        one or more processors and memory coupled to said one or more        processors, said memory storing instructions, which, when        executed by said one or more processors, causes said one or more        processors to perform operations comprising:    -   obtaining one or more first images with a first imaging        modality;    -   processing the first images to calculate, for a plurality of        regions within the first images, an image measure associated        with the suitability of a second imaging modality;    -   in the event that the image measure for one or more regions lies        within a pre-selected range, acquiring one or more second images        with the second imaging modality.

In another aspect, there is provided a method of performing adaptiveillumination while performing optical imaging during a medicalprocedure, the method comprising:

-   -   determining the field of view of an optical imaging device        employed during the medical procedure;    -   determining configuration parameters of an illumination source        for improving the homogeneity of illumination within the field        of view;    -   configuring the illumination source according to the        configuration parameters.

In another aspect, there is provided a system for performing adaptiveillumination while performing optical imaging during a medicalprocedure, comprising:

-   -   a control and processing system interfaced with the optical        imaging device and the illumination source, said control and        processing system comprising one or more processors and memory        coupled to said one or more processors, said memory storing        instructions, which, when executed by said one or more        processors, causes said one or more processors to perform        operations comprising:    -   determining the field of view of an optical imaging device        employed during the medical procedure;    -   determining configuration parameters of an illumination source        for improving the homogeneity of illumination within the field        of view;    -   configuring the illumination source according to the        configuration parameters.

A further understanding of the functional and advantageous aspects ofthe disclosure can be realized by reference to the following detaileddescription and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described, by way of example only, withreference to the drawings, in which:

FIG. 1 illustrates an example automated system for a minimally-invasiveneurological surgical procedure employing an access port.

FIG. 2 shows a human brain into which an access port has been inserted,establishing an open conduit for providing access to tissue within thebrain.

FIG. 3 is flow chart illustrating an example method of intraoperativelydetermining configuration parameters for a device based on theintraoperative identification of a medical instrument.

FIG. 4 is a flow chart illustrating an example method of identifying amedical instrument.

FIG. 5A illustrates an example surgical system including an opticalimaging system and associated control system, where one or morecomponents of the optical system are adaptively configured based on theidentification of a medical instrument.

FIG. 5B shows the re-configuration of the system shown in FIG. 5A afterthe identification of a different medical instrument within the surgicalfield.

FIG. 5C shows a table providing example configuration data associatingconfiguration parameters for a camera with the identity of variousmedical instruments.

FIG. 5D shows a table providing example configuration data associatingconfiguration parameters for an imaging optics assembly with theidentity of various medical instruments.

FIG. 5E shows a table providing example configuration data associatingconfiguration parameters for an illuminator with the identity of variousmedical instruments.

FIG. 5F shows a table providing example configuration data associatingconfiguration parameters for illuminator focusing optics with theidentity of various medical instruments.

FIG. 5G shows a table providing example configuration data associatingconfiguration parameters for a camera with a ranked list of medicalinstruments.

FIG. 5H shows a table providing example configuration data associatingconfiguration parameters for a camera with the identity of variousmedical instruments, including configuration parameters associated withthe absence of a detected medical instrument.

FIG. 5I shows a table providing example configuration data associatingconfiguration parameters for a camera with the identity of variousmedical instruments, where the configuration parameters are furtherassociated with the type of medical procedure being performed.

FIG. 5J shows a table providing example configuration data associatingconfiguration parameters for a camera with the identity of variousmedical instruments, where the configuration parameters are furtherassociated with the phase of the medical procedure.

FIG. 5K shows a table providing example configuration data associatingconfiguration parameters for a robotic arm with the identity of variousmedical instruments.

FIG. 5L shows a table providing example configuration data associatingconfiguration parameters for a user interface with the identity ofvarious medical instruments.

FIG. 6A shows an example user interface prior to the identification of amedical instrument.

FIG. 6B shows the reconfiguration of the example user interface fromFIG. 6A, after the identification of a medical instrument.

FIGS. 7A and 7B illustrate an example embodiment in which newconfiguration parameters are provided for intraoperatively changing theconfiguration of an optical system when an access port is identified,where FIG. 7A shows the configuration of the system prior to theidentification of the access port, and FIG. 7B shows the configurationof the system after the identification of the access port.

FIG. 7C demonstrates an example implementation of the use of ray tracingto calculate a configuration parameter specifying the working distanceof the illuminators in order to achieve a pre-selected level ofillumination homogeneity.

FIG. 8 shows a block diagram of an example system configuration,including the control and processing unit and a number of externalcomponents.

FIG. 9 is a flow chart illustrating an example method ofintraoperatively configuring one or more devices based on theimage-based detection of a context measure associated with a medicalprocedure.

FIG. 10A is a flow chart illustrating an example method of performingtissue identification via hyperspectral imaging.

FIG. 10B shows a table providing an example of configuration data thatassociates configuration parameters for illuminators with one or moretissue types.

FIG. 11A is a flow chart illustrating an example method of identifyingthe phase of a medical procedure based on hyperspectral imaging.

FIG. 11B shows a table providing an example of configuration data thatassociates configuration parameters for a camera with the phase of amedical procedure.

FIG. 12 shows an example access port having calibration features ortargets that can be imaged and analyzed to automatically obtain one ormore measures associated with color balance, white balance, dynamicrange and illumination uniformity.

FIG. 13A a flow chart illustrating an example method of controlling asecond imaging modality based on intermittent sampling and processing ofimage data from the second imaging modality while obtaining images usinga first imaging modality

FIG. 13B is a flow chart illustrating an example adaptive andinteroperative method of controlling the acquisition rate of imagespertaining to an imaging modality.

FIG. 14 is a flow chart illustrating an example implementation of amethod of intraoperatively and adaptively controlling the acquisition ofimages from white light and hyperspectral imaging modalities.

FIG. 15 shows a flow chart illustrating an example method of adaptivelycontrolling the use of fluorescence imaging based on the intermittentsampling of and processing of fluorescence images.

FIG. 16 is a flow chart illustrating an example method in which imagesfrom a first imaging modality may be obtained and processed in order totrigger the acquisition of images from a second imaging modality.

FIG. 17 is a flow chart illustrating an example method in which whitelight images are obtained and processed in order to trigger theacquisition of images using a hyperspectral imaging modality.

FIG. 18 is a flow chart illustrating an example method in whichhyperspectral images are obtained and processed in order to trigger theacquisition of images using a near-infrared imaging modality.

FIG. 19 is a diagram depicting a mock head, mock brain, and mock headholder of a patient with a tracking marker reference.

FIG. 20 is a diagram depicting medical instruments with correspondingtracking templates and optical tracking markers.

FIG. 21 is a flowchart describing the phases of a port basedneurosurgery.

FIG. 22 is a flow chart describing the analysis of images and activationof polarization imaging.

FIG. 23 is a flow chart describing the analysis of spatial data and theactivation of polarization imaging.

FIG. 24 is a flow chart describing the analysis of images and activationof near infrared imaging.

DETAILED DESCRIPTION

Various embodiments and aspects of the disclosure will be described withreference to details discussed below. The following description anddrawings are illustrative of the disclosure and are not to be construedas limiting the disclosure. Numerous specific details are described toprovide a thorough understanding of various embodiments of the presentdisclosure. However, in certain instances, well-known or conventionaldetails are not described in order to provide a concise discussion ofembodiments of the present disclosure.

As used herein, the terms “comprises” and “comprising” are to beconstrued as being inclusive and open ended, and not exclusive.Specifically, when used in the specification and claims, the terms“comprises” and “comprising” and variations thereof mean the specifiedfeatures, steps or components are included. These terms are not to beinterpreted to exclude the presence of other features, steps orcomponents.

As used herein, the term “exemplary” means “serving as an example,instance, or illustration,” and should not be construed as preferred oradvantageous over other configurations disclosed herein.

As used herein, the terms “about” and “approximately” are meant to covervariations that may exist in the upper and lower limits of the ranges ofvalues, such as variations in properties, parameters, and dimensions. Inone non-limiting example, the terms “about” and “approximately” meanplus or minus 10 percent or less.

Unless defined otherwise, all technical and scientific terms used hereinare intended to have the same meaning as commonly understood to one ofordinary skill in the art. Unless otherwise indicated, such as throughcontext, as used herein, the following terms are intended to have thefollowing meanings:

As used herein, the phrase “medical instrument” refers to a tool,instrument, or other implement employed during a medical procedure. Amedical instrument may be provided in various forms, such as, but notlimited to, a handheld or robotically positioned tool, or a componentthat is attached to, or inserted into, a patient during a surgical ormedical procedure. Non-limiting examples of medical instruments include,but are not limited to, scalpels, bi-polar devices, suction devices,cutting devices, clamping devices, access ports, Imaging devices,spectroscopy devices, and suturing tools.

As used herein, the phrase “operator” refers to a user, medicalpractitioner, surgeon, imaging technician, or other individual or groupof individuals involved in operating medical instruments, devices andequipment during a medical procedure.

As used herein, the phrase “tracking system” refers to a systemconfigured to track the position and/or orientation of one or moreobjects, such as locations on a patient and/or surgical instruments. Insome embodiments, the tracking system may be configured to track theposition and/or orientation of an imaging device (such as an opticalcamera). A tracking system may also be employed to track the positionand/or orientation of an access port or other component that is attachedto, or inserted into, a patient or subject. In one example, a trackingsystem may employ a pair of infrared cameras to track the position andorientation of active or passive infrared spheres (fiducials) attachedto one or more objects, such as the Polaris® system from NDI.

As used herein, the phrase “navigation system” refers to a system thatprocesses and spatially registers pre-operative image data to aninteroperative reference frame, and displays the position andorientation of one or more tracked items relative to the pre-operativeimage data. A navigation system may interface with, or include, atracking system, in order to track the items. In some exampleimplementations, hardware associated with the navigation system mayinclude a computer system, a display, and a tracking system.

As used herein, the phrase “phase of the medical procedure” refers to agiven step, or set of sequential steps, within a medical procedure. Inanother example, a phase of a medical procedure need not be a given stepor set of sequential steps in a procedure, but may relate to the use ofa specific tool or set of tools within a given step of a medicalprocedure.

As used herein the phrase “intraoperative” refers to an action, process,method, event or step that occurs or is carried out during at least aportion of a medical procedure. Intraoperative, as defined herein, isnot limited to surgical procedures, and may refer to other types ofmedical procedures, such as diagnostic and therapeutic procedures.

As used herein, the phrase “access port” refers to a cannula, conduit,sheath, port, tube, or other structure that is insertable into asubject, in order to provide access to internal tissue, organs, or otherbiological substances. In some embodiments, an access port may directlyexpose internal tissue, for example, via an opening or aperture at adistal end thereof, and/or via an opening or aperture at an intermediatelocation along a length thereof. In other embodiments, an access portmay provide indirect access, via one or more surfaces that aretransparent, or partially transparent, to one or more forms of energy orradiation, such as, but not limited to, electromagnetic waves andacoustic waves.

Presently, many devices that are employed during a medical procedure arecontrolled independently of the actions that are being performed duringthe procedure. For example, lighting systems typically operate in anindependent manner without receiving any controlling input that isassociated with the current phase or context of a medical procedure. Inanother example, external imaging devices, such as an endoscopes, areusually independently activated and controlled by an operator while amedical procedure is being performed.

Several embodiments of the present disclosure provide systems andmethods in which devices that are employed during a medical procedureare adaptively and dynamically configured and/or controlled during themedical procedure, based on input or feedback that is associated withthe current phase or context of the medical procedure. In some exampleembodiments, the input is obtained via the identification of one or moremedical instruments present within a region of interest (such as asurgical field), and this input may be employed to determineconfiguration parameters for configuring the device. In other exampleembodiments, the input may be based on the image-based detection of ameasure associated with the phase or context of the medical procedure,and this input may be employed to adaptively control the device based onthe inferred context or phase of the medical procedure. In still otherembodiments, images from one imaging modality may be employed toadaptively switch to another imaging modality. These and other exampleembodiments are described in detail below.

FIG. 1 illustrates an example automated system for performing variousembodiments of the present disclosure, providing a non-limiting examplepertaining to a minimally-invasive neurological surgical procedureemploying an access port. The example automated system includes anautomated robotic arm 105, which supports an optical video scope 110(and associated illumination), video display 115 for displaying a videoimage from optical video scope 110, navigation display 116 for providinga navigation user interface, a tracking device 120 for tracking variousmedical instruments within the surgical field, and a control andprocessing unit 400 for controlling various devices (such as the roboticarm 105) and providing surgical navigation. A patient's head is held inplace by a head holder 125, and inserted into the head is an access port130 and introducer 135 (having fiducial markers attached thereto).Introducer 135 is shown received within access port 130 in the figure,and is tracked using tracking system 120.

The position of the patient may be initially determined and/orcontinuously tracked intraoperatively by tracking system 120. A set ofpreoperative images associated with the anatomy of interest of thepatient may be obtained prior to surgery. These images may beintraoperatively registered to the patient, for example, by way ofsurface matching, sets of known touch points (tip of nose, temple, ears)and/or fiduciary markings that can be identified on the patient and inthe associated images. These points or surfaces are registered to thetracking coordinate frame through a defined registration process. Onceregistered, medical instruments, and the associated patient images canbe tracked in real-time, and shown in various manners on a computermonitor.

The example automated system illustrated in FIG. 1 is configured for theapplication of minimally invasive brain surgery, using an access port toprovide a conduit within the head, allowing access to internal braintissue for surgical, therapeutic, or diagnostic applications. The figureshows an intracranial access port which may be employed in neurologicalprocedures in order to provide access to internal tissue pathologies,such as tumors. One example of an intracranial access port is theBrainPath surgical access port provided by NICO, which may be insertedinto the brain via an obturator (introducer) with an atraumatic tip inthe brain. Such an access port may be employed during a surgicalprocedure, by inserting the access port, via the obturator that isreceived within the access port to access an internal surgical site.

FIG. 2 illustrates the use of an access port, showing a human brain 140into which an access port 130 has been inserted, thereby establishing anopen conduit providing access to tissue deep within the brain. Surgicalinstruments may then be inserted within the lumen of the access port inorder to perform surgical, diagnostic or therapeutic procedures, such asresecting tumors as necessary. This approach allows a surgeon, orrobotic surgical system, to perform a surgical procedure involving tumorresection in which the residual tumor remaining after is minimized,while also minimizing the trauma to the intact white and grey matter ofthe brain. In such procedures, trauma may occur, for example, due tocontact with the access port, stress to the brain matter, unintentionalimpact with surgical devices, and/or accidental resection of healthytissue. For example, access port based procedures may be employed forother surgical interventions for other anatomical regions, such as, butnot limited to, spine, knee and any other region of the body that willbenefit from the use of an access port or small orifice to access theinterior of the human body.

Referring again to FIG. 1, in order to introduce the access port 130into the brain, introducer 135 with an atraumatic tip may be positionedwithin the access port and employed to position the access portionwithin the head. As noted above, introducer 135 (or access port 130) mayinclude fiducials for tracking. These fiducials may be passive or activefiducials, such as reflective spheres for passive infrared detection viaan optical camera, or, for example, pick-up coils in the case of anelectromagnetic tracking system. The fiducials are detected by trackingsystem 120 and their respective positions are inferred by trackingsoftware (which may reside within tracking system 120, or may reside,for example, within control and processing unit 400).

Once access port 130 is inserted into the brain, introducer 135 may beremoved to allow for access to the tissue through the central opening ofaccess port 130. However, once introducer 135 is removed, access port130 can no longer be directly tracked in real time (according to theexample embodiment shown in FIG. 1 in which no fiducials are attached toaccess port 130). In order to track the position and orientation ofaccess port 130, it may be indirectly and intermittently tracked by apointer tool having fiducials that are detectable by tracking system120.

Although the example system described in FIG. 1 relates to aneurosurgical procedure, it will be understood that the systems andmethods described herein are not intended to be limited to neurosurgicalprocedures or port-based procedures, and may be employed for a widerange of medical procedures. Examples of other types of medicalprocedures including orthopedic, trauma, gastrological, cardiac,gynecological, abdominal, ENT, oral and maxillofacial, urological,dental, and other surgical, diagnostic or therapeutic medicalprocedures. It is further noted that while many of the exampleembodiments described herein employ external imaging, such as imagingwith an external video scope, it will be understood that variousinternal imaging devices, such as endoscopic or catheter imagingdevices, may additionally or alternatively be employed. It is furthernoted that embodiments of the present disclosure may be employed withinor adapted to procedures employing telesurgical or shared-controlsystems.

In many of the example embodiments described below, each medicalinstrument that is to be tracked may have a fiducial attached thereto(e.g. passive or active fiducial markers, such as reflective spheres oractive LED lighting emitted from at least 3 points on a device) so thatthe position and orientation of the instrument can be determined. In oneexample implementation, the fiducial markers may be employed todetermine a reference position on medical instrument (such as a centralpoint), and an axis of the medical instrument (such as a longitudinalaxis of a tool).

In some embodiments, the identification of one or more medicalinstruments, as described above, may be employed to adaptively ordynamically provide configuration parameters for controlling one or moredevices that are employed within a medical procedure. A “configurationparameter”, as used herein, refers to a parameter for adjusting theconfiguration of a device, as opposed to an instruction or signal thatis employed to activate (turn on) the device.

This method of actively determining configuration parameters for adevice is to be contrasted with methods known in the art in which adevice is activated, or powered on, based on the identification of asurgical tool. Such an “activating” method is disclosed in US PatentApplication Publication No. US2014/0006049 (de la Barrera et al.). Oneexample disclosed in US2014/0006049 involves automatically activating asuction device when the suction device is identified near a surgicalincision.

The present inventors have found that it is often insufficient to merelyactivate a device based on the identification of a tool. Specifically,the inventors have found that many devices that are employed during amedical procedure are not configured merely in a simple binary manner,and have more complex states than simply being either “on” or “off”.Many devices that are employed during a medical procedure are operatedaccording to a set of configuration parameters. Accordingly, in someexample embodiments of the present disclosure, a device employed duringa medical procedure may be dynamically controlled via the selection ofconfiguration parameters that are intraoperatively determined (e.g.customized) based on the identification of a medical instrument. Asnoted above, in some embodiments, the medical instrument need not betracked by the tracking system, provided that the medical instrument canbe identified.

Referring now to FIG. 3, a flow chart is provided that illustrates anexample method of intraoperatively determining configuration parametersfor a device based on the intraoperative identification of a medicalinstrument. At step 200, a medical instrument is intraoperativelyidentified, as described below. The identity of the medical instrumentis then employed to determine customized configuration parameters foradaptively and intraoperatively configuring at least one device. Theconfiguration parameters may be obtained from pre-selected configurationdata associating customized configuration parameters for one or moredevices with the identities of different medical instruments. In step210, the customized configuration parameters are employed to adaptivelyconfigure the device during the medical procedure. For example, theconfiguration parameters can be employed to generate suitable commandsthat are transmitted from control and processing unit 400 to a device,to be executed by the device firmware for reconfiguring the device.

As illustrated in FIG. 4, the identification of the medical instrumentmay be performed, for example, as follows. In step 220, one or moresignals associated with the medical instrument are detected. The signalsmay be, for example, images showing the medical instrument and/orfiducial markers attached to the medical instrument, optical signalsemitted from markers attached to the medical instrument (e.g. pulsesfrom an active fiducial marker), and RFID signals emitted from RFID tagsattached to the medical instrument. These signals may then be processedto obtain one or more identification measures associated with theidentity of the medical instrument, such as an RFID tag value, a codeassociated with an optical pulse sequence, as shown in step 225.

The identity of the medical instrument is then obtained, as shown instep 230, by comparing the identification measures with pre-selectedidentification data, where the pre-selected identification dataassociates the identities of a plurality of medical instruments withvarious measures. The identification data may be provided in the form ofa database, look-up table, or other data structure that may be accessedby control and processing unit 400 to identify the medical instrument.

In one example embodiment, the identity of a medical instrument may bedetermined based on a unique spatial arrangement of fiducial markersattached to the medical instrument. In such a case, the detected signalmay be stereoscopic images obtained by the tracking system, and thestereoscopic images may then be processed determine the positions ofpassive fiducial markers attached to the medical instrument. Themeasures associated with the identity of the medical instrument mayobtained as the relative distances (and/or angles) between the fiducialmarkers. These measures may be compared to a look-up table thatcorrelates the identities of different medical instruments withdifferent relative marker distances or angles. The table entry havingrelative marker distances or angles that matches the calculated measuresprovides a determination of the identity of the medical instrument.

In another example implementation, a medical instrument may beidentified based on marker geometry (e.g. using passive opticalsystems). For example, a medical instrument may be identified by thesize and/or shape of the fiducial markers. In such a case, theidentification data may correlate the geometry of the fiducial markerswith the identities of various medical instruments.

In other example implementations, a medical instrument may employ activefiducials, and the characteristics of the active fiducials may beemployed to identify the medical instrument. For example, activefiducials may be provided in the form of pulsed optical devices, such aslight emitting diodes, where the pulsed pattern may be employed toidentify a medical instrument. The associated identification data mayinclude information, for a plurality of different medical instruments,correlating the characteristic of the active fiducials (e.g. a pulsesequence) with the identity of each medical instrument.

In another example, glyphs, barcodes, and other symbolic, graphic ortextual markers may be employed to identify a medical instrument. Forexample, one or two-dimensional barcodes may be employed to providedetectable information identifying a medical instrument. In such a case,the identification data may correlate the graphical symbol, or a code orother information extractable from the graphical symbol, with theidentities of various medical instruments.

In other example embodiments, electromagnetic and/or radiofrequency (RF)systems may be employed to identify and/or differentiate medicalinstruments based on an electrical trigger rather than geometryconstraints. For example, a medical instrument may be identified basedon radio-frequency identification (RFID) signals.

These tools may be tracked relative to a second set of reflective oractive points, or some other known point with respect to the patient'sanatomy and the tracking system (such as the tracking detector itself),which define the system reference frame. For example, a rigid piece ofstainless steel (patient reference) may be rigidly attached to theHeadholder [125] and have an arrangement of passive reflective fiducialmarkers as shown as (1900) in FIG. 19. The tracking camera [120] will,once having identified the location of this patient reference, establisha co-ordinate system with the origin based at this patient reference(1900) and then report the position and orientation of other tools withrespect to this co-ordinate system.

Although some of the preceding examples pertain to fiducial markers thatare capable of performing a dual role of position/alignment sensing andinstrument identification, it will be understood that these roles may bedecoupled in some embodiments, such that one or more fiducial markersare provided for the tracking of a medical instrument, and one or moreadditional identifying markers are provided for identifying a medicalinstrument.

In such embodiments, the system may employ one or more additionalcameras in order to image the identifying markers, and the system mayinclude an image processing module to process the images to obtain oneor more measures associated with the identity of the medical instrument.In such an embodiment, the additional cameras may be spatiallyregistered to the reference frame of the tracking system, in order tocorrelate the position of an identified marker within the image with theposition of medical device tracked by the tracking system, which may beperformed by control and processing unit 400.

For example, in one illustrative implementation, a medical instrumentmay be provided with a set of passive infrared fiducial markers fortracking, and one or more two-dimensional barcodes for instrumentidentification. In some embodiments, two or more identifying markers maybe positioned at a plurality of locations to avoid shadowing andline-of-sight issues. It will be understood, however, that unliketracking, which requires continuous detection of fiducial markers,identification of a medical instrument need only be performed once whenthe medical instrument enters the region of interest or surgical field,because once a medical instrument is identified, its identity can becontinuously associated with its tracked location through its uniquetracking marker.

In another example embodiment, the images obtained by an additionalcamera may be processed to identify a medical instrument based on aknown instrument shape, or based on known markings or features that maybe imaged, for example the templates (2040 and 2030) located on medicalinstruments (2020 and 2040 respectively) as shown in FIG. 20. Thesetemplates or instrument shapes may be identified using any of themethods described in the paper [Monocular Model-Based 3D Tracking ofRigid Objects: A Survey, section 4].

Although the preceding examples disclose embodiments in which a medicalinstrument is both tracked and identified, it will be understood that insome embodiments, identification of a medical instrument may beperformed without tracking the medical instrument. For example, somemedical instruments may be employed without fiducial markers fortracking. Such non-tracked medical instruments may nonetheless beidentified by any of the preceding example methods, or any othersuitable identification methods. In another example implementation, amedical instrument may be identified based on input from an operator.

It will be understood that the preceding examples of devices and methodsfor identifying a medical instrument are provided as a non-limiting setof illustrative embodiments, and that additional or alternative devicesor methods may be employed to identify a medical instrument withoutdeparting from the scope of the present disclosure.

Referring again to FIG. 3, after having identified a medical instrumentin step 200, customized configuration parameters are obtained foradaptively configuring one or more devices that are employed during amedical procedure.

The one or more devices for which configuration parameters are providedmay be selected from a wide variety of configurable devices that may beemployed during a medical procedure. For example, a device that isadaptively configured according to the present method may be anothermedical instrument that is not connected or connectable to theidentified medical instrument.

In another example implementation, a device for which configurationparameters are provided, based on the identification of the medicalinstrument, may be an auxiliary device, such as, but not limited to, anillumination device, video and sound recording devices, and imagingdevices.

In one example implementation, the devices for which configurationparameters are provided may include an illumination device and/or anoptical imaging device. For example, an example surgical system mayinclude the optical imaging system, and associated control system, shownin FIG. 5A. The figure shows a subset of the components of a surgicalsystem, and does not show some components in order to simplify theillustration (for example, a tracking system is not shown). The examplesystem includes an optical system 250 including camera 255, imagingoptics assembly 260, illuminators 265, illumination focusing optics 270,and auxiliary imaging modality assembly 275. An image detected by camera255 is displayable on display 115, as illustrated in FIG. 1. Opticalsystem 250 may be supported by robotic arm 105. Imaging optics assembly260 is configured to image with a field of view as shown at 280, whileilluminators 265 and illuminator focusing optics 270 projectillumination beams 285 to form illumination region 290 on tissue surface295.

With regard to illuminators 265, a wide variety of illuminator types maybe employed to provide intraoperative illumination. Examples ofdifferent types of illumination include a monochromatic or narrow bandlight source or laser source, a broadband source (such as a white lightsource) optionally having a spectrally selective element such as anoptical filter, a projector type source (which may optionally beemployed for surgical guidance or for projecting customized lightpatterns onto the surgical field), a polarized light source implementedby polarizing filters, structured light projectors, photo-acousticexcitation lasers, ablation lasers, and therapeutic light sources usedfor photo-activation of therapeutic solutions. Illumination light can belocally generated as shown in the figure, or optionally externallygenerated and directed to optical system 250 using an optical conduitsuch as a fiber optic bundle, a light pipe, or free space delivery. Theillumination may be broad to fill the entire surgical field of view(e.g. at the end of an access port), or focused on a particular point.Each of these light delivery devices can be controlled by control andprocessing unit 400.

As shown in FIG. 5A, control and processing unit 400 may be interfacedwith one or more components of optical system 250 in order todynamically provide configuration parameters based on the intraoperativeidentification of one or more medical instruments. Control andprocessing unit 400 is shown interfaced with camera 255, imaging opticsassembly 260, illuminators 265, illumination focusing optics 270, andauxiliary imaging modality assembly 275. Upon detection of a medicalinstrument, the configuration data may be accessed in order to determinecustomized configuration parameters for one or more components of theoptical system, and the customized configuration parameters may beemployed to configure or reconfigure the one or more components.

In the example case illustrated in FIG. 5A, a coarse resection tool (notshown in the figure) has been identified. Customized configurationparameters are obtained for customizing one or more of camera 255,imaging optics assembly 260, illuminators 265, illumination focusingoptics 270, auxiliary imaging modality assembly 275, robotic arm 105,and a user interface displayed on display 115, based on theidentification of the coarse resection tool.

When the coarse resection tool is removed from the surgical field and afine resection tool is brought within the surgical field, the absence ofthe gross section tool and the presence of the fine resection tool isdetected, with the fine resection tool being identified by the system asdescribed above. New customized configuration parameters are obtained,and the optical system 250 is reconfigured as shown in FIG. 5B. In theexample case shown in the figure, configuration parameters for a numberof components have been modified due to the identification of the fineresection device. Specifically, robotic arm 105 has been repositionedaccording to updated configuration parameters to achieve a reducedworking distance; imaging optics assembly has been reconfigured toprovide a reduced field of view 280 and therefore higher magnification;illumination focusing optics 270 have been reconfigured to produce areduced illumination region; and illuminators 265 have been reduced inintensity in order to preserve the intensity of illumination within theillumination region 290.

Additionally, for example, the system may be further reconfigured byproviding configuration parameters for any one of more of room lights(e.g. dimming or increasing brightness), coarse resection toolreconfiguration, fine resection tool reconfiguration, adjustment ofspeed and/or power of the fine resection tool, modifying hangingprotocols displayed on the navigation screen (e.g. display differentsets of images and different views of those images), and adjust theangle or height of the surgical table.

In one embodiment, fine resection tool is tracked by tracking system120, and the customized configuration parameters configure robotic arm105 to be actuated such that the field of view 280 of imaging opticsassembly 260 is actively translated to overlap with the distal tip ofthe fine resection device based on closed-loop feedback from trackingsystem 120.

In one example implementation, control and processing unit 400 may beinterfaced with camera 255 in order to adaptively provide configurationparameters associated with one or more of, but not limited to, imagingframe rate, gain, saturation, shutter speed, ISO, aperture size, on-chipbinning, image size, digital zoom (ROI), and cooling temperature (e.g.if thermo-electric cooling is available). An example of configurationdata that associates configuration parameters for camera 255 with one ormore medical instruments is shown in FIG. 5C.

Control and processing unit 400 may additionally or alternatively beinterfaced with imaging optics assembly 260 in order to provideconfiguration parameters associated with one or more of, but not limitedto, zoom (magnification), focal length, working distance, numericalaperture, polarization sensitivity, attenuation, filter wavelength,depth of field, image stabilization and field of view. For example,imaging optics assembly 260 may include one or more actuators forvarying these settings according to the configuration parameters thatare provided. An example of configuration data that associatesconfiguration parameters for imaging optics assembly 260 with one ormore medical instruments is shown in FIG. 5D.

Control and processing unit 400 may additionally or alternatively beinterfaced with illuminators 265 in order to provide configurationparameters associated with one or more of, but not limited to,illumination intensity, illumination wavelength, illumination angle,pulsed or continuous operation, and number of active illuminators. Forexample, illuminators 265 may include one or more actuators for varyingthe incidence angle of the illumination beams according to theconfiguration parameters that are provided. An example of configurationdata that associates configuration parameters for illuminators 265 withone or more medical instruments is shown in FIG. 5E.

Control and processing unit 400 may additionally or alternatively beinterfaced with illumination focusing optics 270 in order to provideconfiguration parameters associated with one or more of, but not limitedto, focal length, depth of field, illumination spot size, beam shape,working distance, polarization, filter wavelength, and attenuation. Forexample illumination focusing optics 270 may include one or moreactuators for varying these settings according to the configurationparameters that are provided. An example of configuration data thatassociates configuration parameters for illumination focusing optics 270with one or more medical instruments is shown in FIG. 5F.

Control and processing unit 400 may additionally or alternatively beinterfaced with auxiliary imaging modality assembly 275. For example,auxiliary imaging modality assembly 275 may include one or more opticalports, and a mechanism, such as an optical deflection device (e.g. amirror, prism, reflector, filter, pellicle, window, or optical pick-off)that may be selective actuated to deflect the beam path along the portaxis, thereby directing the optical beam to imaging and/or source opticsassociated with another imaging modality. For example, in one exampleimplementation, auxiliary imaging modality assembly 275 may include oneor more ports for selectively employing an additional imaging modalityincluding, but not limited to, fluorescence imaging, infrared imaging,ultraviolet imaging, hyperspectral imaging, optical coherencetomography, polarization-sensitive optical coherence tomography,polarization-sensitive imaging, thermal imaging, photo-acoustic imaging,and Raman imaging. Control and processing unit 400 may thus provide oneor more configuration parameters for selectively configuring the imagingsystem to employ one or more additional or alternative imagingmodalities. Control and processing unit 400 may also provide one or moreconfiguration parameters for selectively configuring the one or moreadditional or alternative imaging modalities.

In some embodiments, one or more external imaging devices may beemployed for multi-modal imaging. For example, multi-modal imaging maybe achieved by way of either direct optical imaging, or using the systemto hold additional imaging probes, such as MRI, US, PET or X-ray (eitherin transmit or receive modes). In some embodiments, the turret ofrobotic arm 105 can be actuated during the procedure to engage differentmodalities, as described above, much in the way multiple tools areselected in a CNC machining system. In other embodiments, multiplemodalities other than optical, for instance ultrasound, MRI, OCT, PET,CT, can be supported by or otherwise interfaced with the automated arm,optionally in addition to one or more optical imaging/detectionmodalities.

In the case of photo-acoustic imaging, laser light is used to excite thetissue, while an ultrasound array positioned in the access port isemployed to collect the emitted ultrasound signal. In addition,different wavelengths or spectral bands of light may be utilized. Forinstance, Raman imaging can be used to investigate the chemicalcomposition of tissue at a specific location of interest, i.e. pointsource imaging. Hyper-spectral imaging can be accomplished by scanning adetector across the region of interest, or collecting a multi-spectraldetector images at a selected location. In one example implementation,the hyperspectral image could be overlaid on video images to providedifferent perspectives of exposed tissue regions. In another exampleembodiment, laser light delivered by an optical device supported by theautomated arm may be employed for the alignment and/or excitation ofphoto-reactive therapeutics. Any or all of the optical imaging modesemployed by a given system embodiment may be accommodated by afiber-optic delivery and receiving bundle that is attached to the turretof robotic arm 105. Alternatively, or in addition, various ports orlight guides may be used to co-align the light delivery or reception.

In an alternate embodiment, optical system 250 can have differentacquisition modes. Some modes are listed as follows but are not limitingto additional modes not listed here. In one mode, images can be acquiredby sweeping through the different image acquisition modes to providemultiple serially obtained (e.g. almost simultaneously obtained) imagesof different types which can be combined into an overlaid representationand displayed to the operator. The multi modal shifting can be achieved,for example, by using a filter wheel on the optical system, allowing theimaging modalities to change as the wheel is turned. It can also beachieved through beam splitting using optical lenses and directing thebeams to different imaging devices.

Although several different components are shown interfaced with controland processing unit 400 in the figure, it is to be understood thatcontrol and processing unit 400 may be interfaced with any component, orany combination of components, and with other components that are notshown.

In an alternate embodiment, the optical system 250, under control ofcontrol and processing system 400, may automatically perform actionssuch as, but not limited to, autofocus of the optical view and autoadjustment of the illumination system for optimal viewing illumination,optimal tissue differentiation, and optimal modal detection. Opticalsystem 250 can achieve these automatic functions through analysis of thevarious images acquired by the system, such as the optical camera imageor others by control and processing system 400. The images can beanalyzed for metrics such as white balance, contrast, and saturation.The metrics can then be processed based on the type of view required,for example when illuminating for tissue differentiation the imagingprocessing method should employ the constraints of the system(geometric, intensity range, etc.) to obtain the illumination intensityand wavelengths which would provide a suitable (e.g. maximal) contrastmetric. Other image analysis that could be done include image sharpnessdetermination and optimization by analyzing specific focal zones.Alternatively, the optical system 250 could adjust zoom and focus bycalculating the working distance between the camera 255 and the surgicalarea of interest by using position and orientation of the surgical tooland position and orientation of the optical system provided by thenavigation system. In the case of port-based surgery, the port could betracked and the zoom and focus be set based on the working distancebetween the camera and bottom of the port. In both of these cases, alookup table could be created that relates working distance to a set ofcamera parameters: zoom, focus, aperture, and iris. This relationshipcould be determined empirically or analytically.

The preceding examples illustrate embodiments in which configurationparameters are provided in a number of data structures pertaining todifferent devices that may be intraoperatively configured based on theidentification of one or more medical instruments. It will be understoodthat the data structures were illustrated separately for heuristicpurposes, and that in other implementations, the two or more datastructures may be combined. For example, a composite data structure maybe formed in which different devices are provided as different columns.

As shown in FIGS. 5C-5F, configuration parameters may be provided forintraoperatively configuring a device based on the identification of asingle medical instrument, or based on the identification of multiplemedical instruments. The example data structures shown in FIGS. 5C-5Fillustrate an example implementation in which configuration parametersare provided for each relevant combination of identified medicaldevices.

In another example implementations, configuration parameters can beprovided for multiple identified medical instruments according to aranked or prioritized list of medical instruments. For example, FIG. 5Gprovides an example implementation of a data structure in whichconfiguration parameters are provided for camera 255 on a unique basisfor each medical instrument that may be identified. Control andprocessing unit 400 may be programmed to interpret the data structure ina ranked configuration. If a single medical instrument is identified, inthe absence of other medical instruments, then the configurationparameter set associated with the single medical instrument is employedto configure camera 255. For example, if only instrument 4 is identifiedat any given time during a medical procedure, in the absence ofinstruments 1-3, 5 and 6, then configuration parameter set 4 is employedto configure camera 255.

However, if two or more medical instruments are intraoperativelyidentified, then configuration parameters associated with the highestranked medical instrument are employed. For example, if medicalinstruments 3 and 5 are identified at a given time during a medicalprocedure, the configuration parameters used to configure camera 255would be configuration parameter set 3, since medical instrument 3outranks medical instrument 5. It will be understood that the specificimplementation illustrated in FIG. 5G is merely one exampleimplementation, and that variants of this embodiment may be performedwithout departing from the scope of the present disclosure. For example,weighting factors or coefficients may be employed to realize a relatedranked or prioritized embodiment.

Although the preceding example implementations illustrate cases in whichconfiguration parameters are provided based on the identification of oneor more medical instruments, it will be understood that theconfiguration data may include a default set of configuration parametersthat may be employed to configure the device when the identifiablemedical instruments are not present, as shown in FIG. 5H. For example,this may be useful for ensuring that a given device reverts to a defaultconfiguration when one or more identified medical instruments areremoved from the surgical field or region of interest.

The example embodiments shown in FIGS. 5C-5H relate the identificationof a medical instrument, by its name or type (e.g. by its clinical,medical, or conventional name), with customized configuration parametersfor adaptively and intraoperatively configuring a device (e.g. camera255). It is to be understood that the identification of an instrument byname or type is but one example implementation of an identificationmethod, and that many other methods may be employed to identify amedical instrument. For example, a medical instrument may be indirectlyidentified with a textual, numeric, symbolic, or alphanumeric instrumentidentifier that is associated with its identity. In such an embodiment,configuration data would include data elements associating one or moreinstrument identifiers with customized configuration parameters, wherethe instrument identifier initially be obtained from the pre-selectedidentification data, as described above.

In some embodiments, the medical instrument may be identified beyond itsname or type. For example, in one embodiment, the instrument may beuniquely identified. In other words, a resection device would not simplybe identified as a generic “resection device”, but would be identifiedwith a unique identifier that is only associated with the specificinstrument that is used, such as an instrument identifier that includesa serial or inventory number associated with the resection device.

In one example implementation, a medical instrument may be identified asbeing associated with a specific operator. For example, a medicalinstrument may be commonly associated with a specific surgeon, and theinstrument identifier may include identifying information associatingthe medical instrument with the specific surgeon. In such an embodiment,the configuration data would correlate the instrument identifier withconfiguration parameters that are preferred by the specific surgeon,such that when the specific medical instrument is used and identified,control and processing unit 400 provides the preferred configurationparameters to the relevant medical device or devices. In one embodiment,biometric sensors may identify the user of the device. The biometricsensors may be integrated into the surgical tool or acquired via aseparate device either attached to the surgeon continuously or separate(e.g. the surgeon activates his/her identity at a computer console).Example biometric identification techniques are: iris scan, fingerprintscan, voice identification, and cardiac patterns.

In one example implementation, the configuration data that is used toassociate the identity of one or more medical devices with customizedconfiguration parameters (for the interoperative configuration of one ormore devices) may depend on the location of the automated system. Forexample, a given medical instrument may be used in two differentoperating rooms, each room having a separate automated system, where thetwo operating rooms are employed for performing different medicalprocedures. In such a case, the configuration data employed by oneautomated system may be different than the configuration data employedby the other automated system, such that the same medical instrument maybe employed in either room, but where different configuration parametersare associated with the medical instrument in each room.

In another example implementation, the configuration data that is usedto associate the identity of one or more medical devices with customizedconfiguration parameters may be further associated with a medicalprocedure. Such an embodiment is illustrated in FIG. 5I, where anadditional column is shown that further associates the configurationparameters with a given medical procedure. According to this exampleembodiment, upon identification of a medical instrument, control andprocessing unit 400, which select the appropriate customizedconfiguration parameters for intraoperatively configuring one or moredevices based on both the identity of the medical instrument and theprocedure that is begin performed. Control and processing unit 400would, in this example embodiment, obtain information identifying theprocedure being performed based on operator input, automated detection(described in detail below), or based on pre-programmed information.

In another example implementation, the configuration data that is usedto associate the identity of one or more medical devices with customizedconfiguration parameters may be further associated with a given step orphase of a medical procedure. Such an embodiment is illustrated in FIG.5J, where an additional column is shown that further associates theconfiguration parameters with a given phase of a medical procedure.According to this example embodiment, upon identification of a medicalinstrument, control and processing unit 400, which select theappropriate customized configuration parameters for intraoperativelyconfiguring one or more devices based on both the identity of themedical instrument and the procedure that is begin performed. Controland processing unit 400 would, in this embodiment, obtain informationidentifying the phase of the procedure being performed based on operatorinput, automated detection (described in detail below), or based onpre-programmed information.

The preceding example embodiments pertaining to the configuration ofdevices based on the identification of one or more medical instrumentswere illustrated within the context of device components associated withan optical system. In one another example implementation, the device forwhich configuration parameters are provided may be a robotic arm that isemployed to position one or more tools or devices, as illustrated inFIG. 5K. Examples of configuration parameters for configuring a roboticarm include, but not limited to, positions and orientations of the arm,motor speeds, safety regions for collision avoidance, stiffness, andhome positions, tip speeds, acceleration profiles, working distanceoffset, position of the arm with respect to the surgeon, updated voicecommand library if the robot is controlled via voice commands, updatedmapping of robot control pedals/buttons, enable/disable of automatictracking modes, thresholds used to trigger automatic movement/alignmentof the robot, safety delay time, movement patterns, the payload weightwill require the arm to compensate accordingly to the new weight,movement for various types of lighting, surgical phase.

In another example implementation, the device for which configurationparameters are provided may be a computer-controlled user interface, asshown in FIG. 5K. For example, as shown in FIG. 5A, display 115 mayprovide a user interface, such as a user interface associated with theoutput from optical system 250, and/or navigation. One or more inputdevices may be employed to allow an operator to interact with the userinterface, and examples are provided below. The configuration parametersthat are associated with the one or more medical instruments may enablethe automated re-configuration of the user interface based on theidentification of one or more medical instruments. Examples ofconfiguration parameters for configuring a user interface include, butnot limited to, displayed windows, displayed icons, colors, one or moredisplayed or displayable menu items, intensity of one or more displayeditems, relative size of displayed windows, relative positioning of oneor more windows, display of images from one or more imaging modalities,selection of imaging modalities for overlay of images from two or moreimaging modalities, and messages or warnings for the operator, selectionof which tracked tool is used to control the view of imaging data,background music selection, default display options (e.g. are augmentedgraphics displayed), user interface skin/theme, mapping of controlbuttons, various data display or analysis options related to particularimaging modalities, and alerts corresponding to identification ofparticular spectral information related to identification of potentialunhealthy tissues.

FIGS. 6A and 6B illustrate an example implementation of a user interfacethat is automatically configurable based on the identification of one ormore medical instruments. FIG. 6A shows the user interface prior to thedetection of the medical instrument (fine resection tool) 601, whileFIG. 6B shows the user interface after the detection of the medicalinstrument 601. In the latter case, control and processing unit 400,having identified the medical instrument, has determined, based on theconfiguration data, configuration parameters for reconfiguring the userinterface. After the identification of the fine resection tool thesystem has switched to hyperspectral mode providing images for bettertissue differentiation. Accordingly the GUI has adapted to thismultimodal imaging by displaying both the visible and hyperspectralimaging simultaneously side by side as opposed to just the single viewdepicted in FIG. 6A. The adapted GUI also provides additional controlbuttons specific to this multimodal display with hyperspectral imaging.For example button 603 which would allow the user to view ahyperspectral spectrum at a specific point on the display of thehyperspectral image if the user chose to do so. Another example is alert605 which changes colour if a particular spectral fingerprint isidentified in the hyperspectral image which is related to an unhealthytissue type for example a tumor tissue spectral fingerprint.

It is further noted that customized configuration parameters may bepre-selected prior to the commencement of the medical procedure, or mayalternatively be provided, or modified, during the medical procedure.For example, a user interface may be provided to facilitate input ormodification of configuration data and/or identification data via asuitable input device. Examples of input devices are provided below. Theuser interface may also be provided to optionally override one or moreconfiguration parameters.

FIGS. 7A and 7B illustrate an example embodiment in which control andprocessing unit 400 is employed to provide new configuration parametersfor intraoperatively changing the configuration of optical system 250when an access port is identified. In FIG. 7A, optical assembly 250 isshown in a configuration to image the top portion of a patient's skull300. This initial configuration may be determined based on configurationparameters obtained based on the identification of one or more medicalinstruments employed for performing a craniotomy (not shown in thefigure), such as a craniotomy drill used specifically for removing asection of the skull for surgical access to the brain, or a scalpel usedto access the skull through the skin.

For example, configuration parameters may be provided that stipulate thediameter of illumination spot 290, and the field of view 280 provided byimaging optics assembly 260. Additional configuration parameters may beprovided to specify a pre-selected working distance between the distalportion of imaging optics assembly 260 and the surface of skull 300, andthese additional configuration parameters may be employed to moverobotic arm 105 to a suitable position for performing the craniotomywhile imaging. In such cases, both optical system 250 and the patient'shead 300 may be spatially referenced to enable the relative positioningof optical system 250. Further examples of configuration parameters thatmay be obtained based on the identification of the medical instrumentsinclude configuration parameters that specify a suitable illuminationintensity, spectral profile, colour, or wavelength. As noted above, theidentification of the medical instruments for performing the craniotomymay also be employed to reconfigure the user interface displayed ondisplay 115.

In the example neurological procedure presently considered, a surgicalaccess port is inserted into the brain after having performed thecraniotomy, as described above. FIG. 7B shows access port 130 insertedinto the patient's head, where distal internal surface 132 of accessport 130 (or the distal aperture of the access port, depending on thetype of access port that is used) is recessed deep within the brain,providing surgical, diagnostic, and/or therapeutic access to braininternal tissue. Control and processing unit 400 identifies access port130, which may be performed, for example, in an automated fashion basedon fiducial markers or other identifying indicia attached to access port130, in an automated fashion via image processing of an image of thesurgical field, or via input from an operator indicating that accessport 130 has been employed.

The configuration data is then processed to obtain configurationparameters that are customized based on the presence of access port 130within the surgical field, and the customized configuration parametersare employed to re-configure one or more components of optical system250. For example, as shown in FIG. 7B, the angle and/or angular width ofillumination beams 285 are modified such that the distal inner surfaceor aperture 132 of access port 130 is illuminated, and the workingdistance and field of view of imaging optics assembly 260 are modifiedfor imaging of the distal surface or aperture 132 of access port 130.Further examples of configuration parameters that may be obtained basedon the identification of the medical instruments include configurationparameters that specify a suitable illumination intensity, spectralprofile, colour, or wavelength for performing one or more port-basedprocedures. The identification of access port 130 may also be employedto reconfigure the user interface displayed on display 115. It is notedthat the configuration of the optical system may be further modified bythe introduction into the surgical field of one or more medicalinstruments having customized configuration parameters associatedtherewith.

In some embodiments, the customized configuration parameters associatedwith the presence of access port 130 may be employed to provide thedelivery of homogenized light through the port to the surgical area ofinterest, thereby potentially permitting improved tissue differentiationbetween healthy and unhealthy brain tissue by potentially reducing glareand reducing shadows which fall on the tissue due to ports. For example,configuration parameters may be provided, on a continuous basis whileaccess port 130 is detected within the surgical field, to activelycontrol the position of robotic arm 105 such that coaxial alignmentbetween the axis of imaging optics assembly 260 and access port 130 ismaintained. These configuration parameters may be computed dynamicallyby control and processing unit 400 based on real-time tracking of theorientation of access port 130 via tracking system 120.

In one example embodiment, configuration parameters associated with theorientation of optical system 250 may be computed in order to achieve adesired level of homogeneity of illumination intensity for illuminationwithin access port 130. For example, optical modelling, such asnon-sequential ray-tracing, may be employed to calculate, for a givenset of optical focusing conditions, a working distance for theilluminators that provides a suitable, or optimal, level of illuminationhomogeneity. The modelling may include both the angular intensitydistribution of the source, and also the optical properties of accessport 130, such that reflections from the port walls may be modelled.

FIG. 7C demonstrates an example implementation of the use of ray tracingto calculate a configuration parameter specifying the working distanceof the illuminators in order to achieve a pre-selected level ofillumination homogeneity. Non-sequential ray tracing software (ZEMAX)was employed to model the illumination intensity distribution at thedistal surface 132 within access port 130, based on the opticalproperties of access port 130 as well as properties of illuminators 265.

Illumination intensity distributions were computed based on fourdifferent models, each having a different illuminator configuration. Thefirst model (910) shows the illumination of the region of interest atthe distal end of an access port 130 using a single illuminator at adistance of 35 cm from the bottom of the access port 130 and offset 16.5mm from the central axis of the access port 130.

The second (920) and third (930) models show illumination of the regionof interest using illumination from two illuminators each. The pairs ofsources in each model are oriented differently with respect to the othermodel. Both models two and three have the same distance and poseparameters as model one relative to the port, 35 cm distance from thebottom of the port and each illuminator offset 16.5 mm from the centralaxis of the access port 130.

The final model (940) shows illumination from two sources with the sameorientation as the sources in the second model (920) relative to theexternal imaging sensor, with the same pose but, a working distance of65 cm. The intensity map on each region of interest (distal end of theport) shown in the figure describes the illumination level, wheremid-range (950) represents the ideal illumination level. As can be seenin FIG. 7C, hot spots (960) exist in models one through three (910, 920,930) which result in heavy glare at those positions and inadequateimaging for the surgeon, while model four (940) provides the optimallighting condition (homogenized and low glare delivery of illumination).Using model four as the optimal pose alignment, the automated mechanicalarm would position the scope to achieve this particular illuminationthereby improving the operating view of the surgeon. The software canthen determine a suitable spatial position and pose of the illuminationsource relative to the target (e.g. the access port) given therestrictions of the system to ensure optimal light delivery through theport to the region of interest.

The illumination source may be also optimally positioned after modellingthe shadow cast by the surgical tools. In other words, the target regionwithin the field of view may be optimally illuminated while avoidingcasting of shadows from the surgical tools. This is possible given thethree-dimensional pose of the surgical tools can be estimated usingfiducial tracking markers placed on the surgical tools.

Referring now to FIG. 8, a block diagram of an example systemconfiguration is shown. The example system includes control andprocessing unit 400 and a number of external components, shown below.

As shown in FIG. 8, in one embodiment, control and processing unit 400may include one or more processors 402, a memory 404, a system bus 406,one or more input/output interfaces 408, and a communications interface410, and storage device 412.

Control and processing unit 400 is interfaced with other externaldevices, such as tracking system 120, data storage 442, and externaluser input and output devices 444, which may include, for example, oneor more of a display, keyboard, mouse, foot pedal, microphone andspeaker. Data storage 442 may be any suitable data storage device, suchas a local or remote computing device (e.g. a computer, hard drive,digital media device, or server) having a database stored thereon. Inthe example shown in FIG. 8, data storage device 442 includesidentification data 450 for identifying one or more medical instruments460 and configuration data 452 that associates customized configurationparameters with one or more medical instruments 460. Data storage device442 may also include preoperative image data 454 and/or medicalprocedure planning data 456. Although data storage device 442 is shownas a single device in FIG. 8, it will be understood that in otherembodiments, data storage device 442 may be provided as multiple storagedevices.

Medical instruments 460 are identifiable by control and processing unit400. Medical instruments 460 may be connected to, and controlled by,control and processing unit 400, or may be operated or otherwiseemployed independent of control and processing unit 400. Tracking system120 may be employed to track one or more of medical instruments andspatial register the one or more tracked medical instruments to anintraoperative reference frame.

Control and processing unit 400 is also interfaced with a number ofconfigurable devices, and may intraoperatively reconfigure one or moreof such devices based on configuration parameter s obtained fromconfiguration data 452. Examples of devices 420, as shown in the figure,include one or more external imaging device 422, one or moreillumination devices 424, robotic arm 105, one or more projectiondevices 428, and one or more displays 115.

Embodiments of the disclosure can be implemented via processor(s) 402and/or memory 404. For example, the functionalities described herein canbe partially implemented via hardware logic in processor 402 andpartially using the instructions stored in memory 404, as one or moreprocessing engines 470. Example processing engines include, but are notlimited to, user interface engine 472, tracking engine 474, motorcontroller 476, image processing engine 478, image registration engine480, procedure planning engine 482, navigation engine 484, and contextanalysis module 486.

It is to be understood that the system is not intended to be limited tothe components shown in the Figure. One or more components control andprocessing 400 may be provided as an external component or device. Inone alternative embodiment, navigation module 484 may be provided as anexternal navigation system that is integrated with control andprocessing unit 400.

Some embodiments may be implemented using processor 402 withoutadditional instructions stored in memory 404. Some embodiments may beimplemented using the instructions stored in memory 404 for execution byone or more general purpose microprocessors. Thus, the disclosure is notlimited to a specific configuration of hardware and/or software.

While some embodiments can be implemented in fully functioning computersand computer systems, various embodiments are capable of beingdistributed as a computing product in a variety of forms and are capableof being applied regardless of the particular type of machine orcomputer readable media used to actually effect the distribution.

At least some aspects disclosed can be embodied, at least in part, insoftware. That is, the techniques may be carried out in a computersystem or other data processing system in response to its processor,such as a microprocessor, executing sequences of instructions containedin a memory, such as ROM, volatile RAM, non-volatile memory, cache or aremote storage device.

A computer readable storage medium can be used to store software anddata which when executed by a data processing system causes the systemto perform various methods. The executable software and data may bestored in various places including for example ROM, volatile RAM,nonvolatile memory and/or cache. Portions of this software and/or datamay be stored in any one of these storage devices.

The preceding example embodiments have described systems and methods inwhich a device is intraoperatively configured based on theidentification of a medical instrument. In other example embodiments,one or more devices may be automatically controlled and/or configured bydetermining one or more context measures associated with a medicalprocedure. A “context measure”, as used herein, refers to an identifier,data element, parameter or other form of information that pertains tothe current state of a medical procedure. In one example, a contextmeasure may describe, identify, or be associated with, the current phaseor step of the medical procedure. In another example, a context measuremay identity the medical procedure, or the type of medical procedure,that is being performed. In another example, a context measure mayidentify the presence of a tissue type during a medical procedure. Inanother example, a context measure may identify the presence of one ormore fluids, such as biological fluids or non-biological fluids (e.g.wash fluids) during the medical procedure, and may further identify thetype of fluid. Each of these examples relate to the image-basedidentification of information pertaining to the context of the medicalprocedure.

An example method of intraoperatively configuring one or more devicesbased on the image-based detection of a context measure associated witha medical procedure is illustrated in the flow chart shown in FIG. 9. Instep 500, one or more images are obtained during the medical procedure.The one or more images are then processed in step 505 to obtain (e.g.calculate) at least one context measure associated with the currentstate of the medical procedure. Examples of various context measures,and methods of obtaining the context measures, are provided below. Theone or more context measures are then employed to obtain one or moreconfiguration parameters for adaptively and intraoperatively configuringat least one device that is employed during the medical procedure.

In one example implementation, optical image analysis is employed toobtain an image measure associated with a tissue type that is exposed orotherwise detectable at the present state of the medical procedure. Oneor more optical images are intraoperatively obtained and processed todetermine the presence of one or more tissue types, and one or morecontext measures are provided that are associated with the detection ofthe one or more tissue types. The one or more context measures are thenused to determine one or more customized configuration parameters.

Various image processing methods may be employed in order to identifythe presence of a tissue type. For example, in one exampleimplementation, tissue identification may be performed via hyperspectraloptical imaging. Hyper-spectral imaging can be accomplished by scanninga spectrally-resolved optical detector across the region of interest andcollecting signals associated with multiple wavelengths at each scanpoint, or collecting a multi-spectral detector images. Various types ofhyperspectral detectors and configurations are known in the art.

FIG. 10A illustrates an example method of performing tissueidentification via hyperspectral imaging. In step 520, hyperspectralimage data is obtained by intraoperatively imaging a region of interest.The hyperspectral image data includes per-pixel spectral data. It willbe understood that the spectral data may be provided as a set of valuesat wavelengths spanning a spectral range of interest that defines adigitized spectrum, or may be provided as discrete values a small set ofwavelengths or wavelength bands, such as red-green-blue intensity data.

In the example method presently considered, the per-pixel spectral datais processed, in step 525, to identify groups of adjacent pixels havinga similar spectral response, thereby spectrally segmenting the imageinto one or more regions of similar spectral content. Pixels withsimilar spectral content may be identified based on the calculation of aspectral similarity measure between adjacent pixels.

For example, a spectral similarity measure for two pixels may becalculated by summing, over all wavelengths, the square of thedifference between the values of the two intensity spectra (optionallyafter initially normalizing the two spectra), and dividing the result bythe square of the average net intensity (summed over all wavelengthvalues) of the two pixels. In such a case, two pixels may be deemed tohave similar spectral responses if their spectral similarity measure isless than a pre-selected threshold value. A suitable threshold value maydepend on the degree of spectral similarity that is sought, which maydepend on the type of medical procedure that is being performed.

In one example implementation, when at least two adjacent pixels arefound to have spectral similarity, the subsequent calculation of thespectral similarity measures for each additional adjacent pixel may beperformed by summing, over all wavelength values, the square of thedifference between the values of (i) the intensity spectrum of theadjacent pixels and (ii) the average intensity spectrum of all otherpixels already having been deemed to be similar, and dividing thisresult by the square of the average intensity of all other pixels havingbeen deemed to be similar. It will be understood that the abovesimilarity method is merely one example method, and that other methodsmay be employed to obtain a similarity measure between pixels.

If a region is identified as having spectral similarity (e.g. similaritywithin the pixels spanning the region), then the average spectralresponse for the pixels within the region may be employed for tissueidentification, as shown at step 530 of FIG. 10. For example, theaverage spectral response for the region may be compared to one or morereference spectra, where the reference spectra are each associated witha given tissue type; for example, particular spectra may be correlatedto tumor tissue or white or grey brain matter. The comparison may beperformed using a similarity measure as described above. In someembodiments, the comparison may be performed only for one or moreregions having a number of pixels that exceeds a minimum value, suchthat tissue identification is only performed for regions beyond athreshold size.

The similarity measure between an average spectral response of anidentified region and a given reference spectrum may be employed toprovide a confidence measure associated with tissue identification,where a higher confidence measure indicates a higher probability thatthe tissue within the identified region corresponds to the tissue typeof the reference spectrum. In one example implementation, a region maybe associated with a tissue type if the confidence measure exceeds apreselected value. In cases in which two or more tissue types areidentified based on having confidence factors that exceed a threshold,then the tissue may be identified based on the tissue type having thehigher confidence measure. The identified tissue type, or an identifierassociated with the identified tissue type, provides a context measureassociated with the current state of the medical procedure.

The image processing steps described above may be performed by an imageprocessing module 478 of control and image processing module 400, asillustrated in FIG. 8. Furthermore, the spectral similarity analysisdescribed above may be performed by context analysis module 486.

It will be understood that the preceding example method of performinghyperspectral tissue identification is but one method of processinghyperspectral image data in order to identify a tissue type, and thatother methods may be employed without departing from the scope of thepresent disclosure. For example, methods of hyperspectral imaging aredisclosed in PCT Patent Application No. PCT/CA2014/______, titled“SURGICAL IMAGING SYSTEMS”, filed on Mar. 14, 2014, which isincorporated herein by reference in its entirety.

Furthermore, it will be understood that tissue identification may beperformed according to other methods than hyperspectral imaging. Forexample, fluorescence spectral image data or Raman spectral image datamay be obtained and processed, in a manner as described above, toidentify one or more tissue types based on similarity to referencespectra. For example, in the case of Raman imaging, tissueidentification may be performed by Raman imaging or by scanning a Ramanpoint probe over multiple locations within a region of interest.

In some embodiments, Raman imaging may be combined with optical imagingin order to identify one or more tissue types.

After obtaining a context measure associated with the current state ofthe medical procedure (in this case, associated with an identifiedtissue type), control and processing unit 400 may be employed to obtainconfiguration parameters for intraoperatively configuring one or moredevices based on the context measure. The configuration parameters areobtained from pre-selected configuration data associating customizedconfiguration parameters for one or more devices with different contextmeasures. The customized configuration parameters are employed toadaptively configure the device during the medical procedure.

This is performed in a manner similar to the preceding embodiment inwhich the configuration parameters are obtained for configuring a devicebased on the identity of medical instruments.

An example of configuration data that associates configurationparameters for illuminators 265 with one or more tissue types is shownin FIG. 10B. For example, one or more tissue types may be associatedwith a pathology, such as a tumor. An example of a configurationparameter for the illumination is the optical illumination spectrum. Thespectrum can be modified such that the illumination light has eithergreater depth penetration or enhances surface contrast based on thescattering and absorption properties of the tissue. This increase ineither light penetration or surface contrast can enable tissue featuresnot visible under white light to be visible, and as such, enable bettertissue identification. Increased light penetration allows forvisualization of subsurface structures embedded in the tissue, whileenhanced surface contrast allows for visualization of fine surfacefeatures. Example illumination spectra parameters are provided in FIG.10B for brain white matter, brain grey matter, and muscle. These threetissues strongly absorb light below 500 nm, 550 nm, and 650 nmrespectively, with lower absorption above these levels.

In another example implementation, optical image analysis, such as theimage analysis methods described above, may be employed for identifyingthe intraoperative presence of one or more types of fluids. For example,the aforementioned example embodiment involving hyperspectral tissueanalysis may be adapted to detect the presence of one or more fluids,based on a comparison to reference spectra from known fluids. Forexample, such a method may be employed to detect the presence ofbiological fluids such as blood, and non-biological fluids such as asaline fluid.

In one example implementation, the intraoperative identification of thepresence of a fluid may be employed to improve imaging performance. Asignificant issue with current surgical optical systems and devices isglare caused by fluids that reflect illumination within a surgicalcavity. The glare can cause imbalance in the dynamic range of an imagingcamera, causing the upper range of the camera's dynamic range to becomesaturated. In addition, glare can cause the illumination intensityacross the frequency spectrum of an imager to be unbalanced, dependingon the illumination and conditions of the medical procedure.Accordingly, in some embodiments, configuration parameters associatedwith the intraoperative presence of a given fluid type may be providedfor intraoperatively reconfiguring one or more components of an opticalimaging system in order to improve imaging conditions. For example,configuration parameters may be provided for modifying and/or improvingimage quality metrics such as color and white balance.

In another example implementation, optical image analysis may beemployed to obtain an image measure that is indirectly associated withthe presence of one or more tissue types, fluids, or other materialproperties. For example, although the preceding example embodimentsemployed a method in which an average spectral response from anidentified region is compared with a set of reference spectra associatedwith tissue or fluids, in other embodiments, the average spectralresponse, or another suitable imaging measure, may be compared withreference spectra that are not directly associated with a given tissueor fluid type.

In one example embodiment, the reference spectra may be associated witha phase or step of the medical procedure. FIG. 11A provides an exampleflow chart illustrating such an embodiment. Steps 540 and 545 may beperformed in a manner similar to steps 520 and 525 of the precedingmethod that was illustrated in FIG. 10A, where image data (such ashyperspectral image data) is intraoperatively acquired and processed toobtain one or more regions with a similar spectral response. In step550, an average spectral response from an identified region is comparedwith reference spectra that are associated with different phases of themedical procedure. For example, one or more reference spectra may beprovided that are associated with a first phase of a medical procedure,while one or more other reference spectra may be provided that areassociated with a second phase of the medical procedure. For example,reference spectra may instead be obtained from previous medicalprocedures, such that a set of reference spectra are provided where eachreference spectra is associated with a given phase of the medicalprocedure. Finally, in step 555, the phase of the medial procedure isobtained based on the similarity between the average spectral responseand the reference spectra (methods of similarity analysis were describedin the preceding example pertaining to tissue type analysis).

In such an example embodiment, the context measure that is employed todetermine one or more configuration parameters for intraoperativeconfiguring a device is the identified phase of the medical procedure.

The reference spectra associated with different phases of the medicalprocedure may be produced according to several different methods. In oneembodiment, the reference spectra may be obtained from previouslyperformed surgical procedures. For example, spectra obtained atdifferent phases of a number of prior medical procedures may be obtainedand employed to provide reference spectra pertaining to the differentphases of the medical procedure. In one example method, multiple spectramay be obtained for each phase of the medical procedure and averaged toproduce a representative average spectra for each phase of the medicalprocedure. In another example implementation, reference spectracorresponding to different phases of the medical procedure may beproduced based on reference spectra of tissues and/or fluids that areexpected to be exposed at different phases of the medical procedure.

An example of configuration data that associates configurationparameters for camera 255 with different phases of the medical procedureis shown in FIG. 11B. During the craniotomy the camera may or may not beutilized by the surgeon given the field of view is normally large enoughfor the surgeon to accurately perform the step without requiringassistance, hence the camera will remain in an neutral state with nozoom. During cannulation the robotic arm holding the camera will orientit in a position and orientation relative to the port to provide a viewof the graduation marks on the introducer. As the introducer withattached port is penetrated into the brain to access the tumor thegraduation marks provide an indication of the depth of the instrument.As such during this stage a view requiring at minimum the ability todecipher the graduation marks on the port is required. During gross andfine resection, the camera is vital to the surgeon as it provides a viewdown the port where the surgeon is performing surgery and the surgeoncannot view well with their own eyes. At these stages the camera will bezoomed to different views such as the entire distal end of the port aswell as the particular tissue being resected during fine resection.

FIG. 21 depicts an exemplary port based surgery flow inclusive of thestages of surgery a surgeon would undertake to complete the procedure.In each stage there are applicable adaptive processes that can run tostreamline the procedure to provide more accurate and time efficientsurgical procedures. These processes can utilize various contextmeasures ranging from some non-limiting examples being intraoperativeimaging, temporal information, spatial positions of objects related tothe surgery, medical instruments being used, intraoperative patientvitals (for example, breathing rate, blood pressure, etc.), etc. Thefollowing paragraph describes various adaptive processes that would berun with respect to the port-based procedure depicted in FIG. 21. Itshould be noted that each stage of the surgery may be identified usingvarious context measures and are also provided below as non-limitingexamples. It should also be noted that similar examples ofconfigurations based on context measures are listed in FIG. 5J.

In the first stage (2100) of the surgery the Craniotomy/Incision stagecan be identified by the navigation system control and processing unitthrough the identification of either a scalpel or neurosurgical drillbeing utilized by the surgeon through methods described herein. Duringthis stage an exemplary adaptive process would involve the UI adjustinguser interface being reconfigured to provide a digital display of thedepth of the drill into the patient's skull as the surgeon is performingthe craniotomy. It should be noted that the depth of the drill can becalculated from the navigation system as it knows the spatial positionand pose of both the drill and skull. Such examples of navigation systemis described in detail in PCT Patent Application No. PCT/CA2014/______,titled “SURGICAL NAVIGATION SYSTEM”, and filed on Mar. 14, 2014, whichis incorporated herein by reference in its entirety.

Once the craniotomy has been completed the next stage (2110) of thesurgery is cannulation (Guidance of Access Port). This stage can beidentified by again recognizing the tools being utilized such as theultrasound used to scan under the surface of the dura and in additionthe introducer which is inserted into the port and used to penetrate thebrain to provide access to the tumor in a non-traumatic manner. Bothtools can be identified using tracking mechanisms as described herein.Another non-limiting context parameter that may be used to identify thestage of the surgery would be the time at which the surgery is occurringrelative to the start of the surgery given this parameter was programmedinto the control and processing unit 400. During the craniotomy, thecontrol and processing unit 400 may be used to maneuver a robotic armmounted with an imaging scope to view the cannulating introducer from anorthogonal angle so the graduation marks located on the introducer maybe read, this adaptive process is described above.

The next stage (2120) in the procedure is gross resection (also referredto as De-bulking of Diseased Tissue in FIG. 21) of the unhealthy tumortissue. At this stage the surgeon is resecting the mass bulk of thetumor from the patient through the access port. Again one contextparameter used by the navigation system control and processing unit 400to identify this stage of the surgery would be the removal of theintroducer instrument from the surgical field and the introduction of aresection tool. This stage (2120) as well as the next one Precision ZoneResection (2140) or more commonly known as fine resection both functionin parallel with the Bleeding Management stages (2130) and (2130).Through a periodic wavelength spectrum analyses of an imaging feed (asdepicted by FIG. 24 and described below in detail) acquired using thevisible imaging device mounted on the robotic arm (used to provide anenhanced view of the distal end of the port where the surgeon isperforming the procedure) a bleed can be identified by the control andprocessing unit 400.

An adaptive response to the identification of blood occluding the viewof the tissue of interest being operated on by the surgeon, would be theoverlay of NIR imaging on the occluded areas of the visible imaging asdepicted by FIG. 23 and described in detail below. It should be notedthat the context parameter used to identify the blood, would be itsvisible wavelength (color) spectrum. During the gross resection stage(2120)), a periodic fluorescence analysis can be performed to theimaging feed acquired using the visible imaging device mounted on theend of a robotic arm as described above. When a particular fluorescencespectrum that is correlated with tumor tissue is determined by theanalysis the system can adaptively configure the imaging device to beginimaging using the fluorescence camera to provide enhanceddifferentiation between the healthy and unhealthy brain tissue. Inaddition the UI may simultaneously be configured to provide a view ofthe fluorescence image beside or overlaid on top of the visible lightimaging.

The next stage in the procedure is fine resection (2140). In this stagethe surgeon begins resecting tumor at the boundaries of the healthytissue. A context parameter which could be potentially used to determinethis stage of the procedure may be the zoom of the visible imagingdevice given that the surgeon indicates that they require some zoomabove a particular threshold. Another context parameter could be whenthe location of the tool is close to the edge of the tumor on therendered MRI scan of the patient. Given that the system knows thelocation of the tool relative to the patient's brain as a result of theregistration of the 3D rendered MRI scan, when the system detects thetool is near a nerve tract, the system can adaptively configure theimaging device to begin acquire polarization sensitive imaging (asdepicted in FIG. 23 and described below in detail).

It should be noted that this imaging provides more structuralinformation than simply visible light imaging as structure can beinferred from the polarized light, the method of which is described in[Wood, M. et al., Polarization Birefringence measurements forcharacterizing the myocardium, including healthy, infarcted, andstem-cell regenerated tissues], J. Biomed. Opt. 15(4), 2010. The contextmeasure in the aforementioned example mentioned would be the location ofthe tracked instrument (in this case a resection device) with respect tothe registered patient brain and rendered MRI scan with DTI data.

During the Therapeutic delivery stage (2160) of the proceduretherapeutic drugs are delivered to the region of interest where thetumor is located. A context measure that can be used to determine thisstage of the procedure could again be the use of a medical instrument,in particular a device used to deliver a therapeutic, such as asolution, to the site of interest. Given the medical instrument beingused to deliver device or potentially an additional instrument alsobeing used at the surgical site of interest is mounted with a pointsource imaging probe able to provide a spectral analysis of a particularpoint at which its aimed, the adaptive system can utilize the spectrumacquired for say an array of points to map onto those points (i.e. onthe imaging feed) the particular spectrums or an analysis of theparticular spectrums assisting the surgeon in identifying where thetherapeutic solution needs to be delivered. In addition the system mayalso be able to identify to the surgeon what particular solution couldbe utilized to most effectively provide therapy to those points if usedin combination with a database system for example the one described inPCT Patent Application No. PCT/CA2014/______, titled “INTRAMODALSYNCHRONIZATION OF SURGICAL DATA” and filed on Mar. 14, 2014, which isincorporated herein by reference in its entirety. The final stage in theprocess (2170) as depicted in FIG. 21 is the closure verification of thecraniotomy after the invasive portion of the procedure has beencompleted.

In the preceding example embodiment, the current phase of a medicalprocedure is identified based on both intraoperative input by theuser(s) and or image analysis, and this context measure is employed todetermine customized configuration parameters for intraoperativelyconfiguring one or more devices. The example implementation describedabove employed spectra image analysis of the surgical field (or of aregion of interest within the surgical field) to extract arepresentative average spectral response, which may be compared withreference spectra associated with different phases of the medicalprocedure. In another example implementation, image analysis may beperformed to identify one or more medical instruments that are beingemployed during a medical procedure, as described in detail above.However, rather than associating the identity of a given medicalinstrument directly with one or more configuration parameters, theidentity of a medical instrument may be associated with a given phase ofthe medical procedure in which the medical instrument is commonlyemployed. Accordingly, the intraoperative identification of one or moremedical instruments, based on image analysis, may be employed to providea context measure identifying the current phase of a medical procedure,and configuration data, such as the example data provided in FIG. 11B,may be provided for the determination of one or more configurationparameters associated with the identified phase of the medicalprocedure.

In some embodiments, as described above, optical imaging may beperformed to determine one or more context measures associated with thepresent state of the medical procedure. For example, optical imaging maybe employed using one or more spectral regions including ultraviolet,visible, and infrared. In another example implementation, fluorescenceimaging may be employed. Other examples of optical imaging modalitiesinclude polarization sensitive imaging, hyperspectral imaging, opticalcoherence imaging, and polarization-sensitive optical coherence imaging,and Raman imaging.

Although the preceding examples describe methods in which one or morecontext parameters are obtained based on intraoperative optical imaging,it will be understood that intraoperative imaging may be performed usingany imaging modality, including, but not limited to, intraoperativemagnetic resonance imaging, intraoperative ultrasound imaging,intraoperative photoacoustic imaging, intraoperative CT, andintraoperative PET.

In one example, by using a calibration features or targets 134 on theaccess port 130, as shown in FIG. 12, and using known properties of theoptical system, intraoperative images containing the calibrationfeatures can be analyzed to automatically obtain a measures associatedwith color balance, white balance, dynamic range and illuminationuniformity (spatial uniformity). FIG. 12 depicts several calibrationfeatures which can be explained as follows. Item (120) is a whitebalance feature in which the processing system analyzes the image anduses this feature as the “true” white color, it can then adjust itsconfiguration parameters such as its color mapping to confirm that thewhite it depicts is the same white as the calibration feature. Item(122) is a grey scale balance calibration feature used in a similarmanner to the one described above for adjusting the imaging deviceconfiguration to match this grey balance range. Item (134) is an RGBcolor balance calibration feature. The imaging device when oriented toview down the port to the distal end can use these calibration featuresin the imaging focus periphery to obtain the optimal image for thesurgery. In another embodiment the calibration features may be orientedwithin the opening of the port on the sidewalls. This embodiment mayprovide better calibration of the imaging device to match it with theinterior of the port. Several published algorithms may be employed toautomatically adjust these image characteristics. For example, thealgorithm published by Jun-yan Huo et. al. (“Robust automatic whitebalance algorithm using gray color points in images,” IEEE Transactionson Consumer Electronics, Vol. 52, No. 2, May 2006) may be employed toachieve automatic white balance of the captured video data. In otherembodiments of the present disclosure, systems and methods are providedfor adaptively and intraoperatively controlling multiple imagingmodalities. An automated system employed during a medical procedure,such as the system shown in FIG. 5A, may include multiple imagingmodalities that may be selectively controlled during a medicalprocedure.

In the example system shown in FIG. 5A, optical system 250 includes aprimary optical imaging modality provided by camera 255, but alsoincludes auxiliary imaging modality assembly 275. As noted above,auxiliary imaging modality assembly 275 may include one or more opticalports, and a mechanism, such as optical deflection device (e.g. amirror, prism, reflector, filter, pellicle, window, or optical pick-off)that may be selective actuated to deflect the beam path along the portaxis, thereby directing the optical beam to imaging and/or source opticsassociated with another imaging modality.

For example, in one example implementation, auxiliary imaging modalityassembly 275 may include one or more ports for selectively employing anadditional imaging modality including, but not limited to, fluorescenceimaging, hyperspectral imaging, optical coherence tomography,polarization-sensitive optical coherence tomography,polarization-sensitive imaging, and Raman imaging. Control andprocessing unit 400 may thus provide one or more configurationparameters for selectively configuring the imaging system to employ oneor more additional or alternative imaging modalities. Control andprocessing unit 400 may also provide one or more configurationparameters for selectively configuring the one or more additional oralternative imaging devices that employ other imaging modalities.

According to one example embodiment, at least two imaging modalities arecontrollable by control and processing unit 400, such that they may beselectively employed during a medical procedure. FIG. 13A provides aflow chart illustrating an example method of controlling a secondimaging modality based on intermittent sampling and processing of imagedata from the second imaging modality while obtaining images using afirst imaging modality. Such an embodiment provides intelligent andcontextually relevant control of the second imaging modality relative tothe first imaging modality.

It will be understood that the first imaging modality and the secondimaging modality need not be associated with two separate imagingdevices, and that in some example implementations, an imaging devicethat is initially configured to obtain images using a first imagingmodality may be adaptively and dynamically configured to obtain imageswith a second imaging modality by modifying the imaging device. Forexample, an optical imaging device may be dynamically switched tofluorescence mode via the introduction of one or more filters into theoptical beam path. In another example, an intraoperative magneticresonance imaging system may be dynamically modified to switch betweendifferent modes of operation (e.g. T1 vs. T2 weighted images) viachanges to the transmit and receive sequence.

As shown in FIG. 13A, images from a first imaging modality areintraoperatively obtained. For example, the first imaging modality mayemploy visible imaging using white light illumination. While obtainingthe images from the first imaging modality, a second imaging modality isintermittently employed in order to obtain images, as indicated at step605. For example, the second imaging modality may employ another type ofoptical imaging, such as fluorescence imaging. During this phase, theacquisition rate of the images from the second imaging modality may belower than the acquisition rate of the images from the first imagingmodality. In some non-limiting example implementations, the initialratio of the acquisition rate of the first imaging modality to that ofthe second imaging modality may be greater than or equal toapproximately 2, 5, 10, 50, 100, 10³, 10⁴, 10⁵ or 10⁶.

The images from the second imaging modality are processed in order toobtain an image measure that is employed to determine whether or not tocontinue imaging with the second imaging modality. Various examples ofimage measures, and methods of calculating such image measures, aredescribed in detail below. As shown in step 615, the image measure iscompared with a pre-selected criterion (or criteria) in order todetermine whether or not to continue imaging with the second imagingmodality. In the event that the image measure meets the pre-selectedcriterion, and it is thus determined that it would be suitable tocontinue imaging with the second imaging modality, the acquisition rateof images from the second imaging modality is increased, as shown atstep 620. On the other hand, if the image measure does not meet thepre-selected criterion in step 615, one or more additional images areobtained using the second imaging modality, and the assessment isrepeated until the pre-selected criterion is met.

The image measure that is obtained to determine whether or not toincrease the acquisition rate of the second imaging modality can beobtained according to a wide range of methods. In one exampleembodiment, the image measure may be associated with a performancemeasure of the second imaging modality. For example, the image measuremay involve a determination of a measure of signal-to-noise ratio ofimaging with the second imaging modality, such that when the imagingmeasure associated with the signal-to-noise ratio exceeds thepre-selected criterion in step 615, the acquisition rate of the secondimaging modality is increased. Another example of a performance-relatedimage measure is a measure of the intensity of the signal that isobtained with the second imaging modality. Yet another example of aperformance-related image measure is the amount of signal within a givenfrequency range or spectral range. These performance measures may beevaluated on a global basis using one or more statistical measures, ormay be evaluated on a local or regional basis. For example, thepre-selected criterion evaluated in step 615 may require that a givenperformance threshold is satisfied by a pre-selected fraction of thepixels forming the image obtained via the second imaging modality. It isfurther noted that a plurality of images may be obtained from the secondimaging modality, and an image measure may be obtained by processing theplurality of images (for example, via averaging the images).

It will be understood that while the preceding paragraphs describe theuse of a single image measure, multiple image measures, and associatedcriterion, may be processed in order to determine whether or not toincrease the acquisition rate of the second imaging modality.

In one example implementation, when the acquisition rate of the secondimaging modality is increased in step 620, the acquisition rate of thefirst imaging modality may be reduced. In another exampleimplementation, when the acquisition rate of the second imaging modalityis increased in step 620, the acquisition rate of the first imagingmodality may be maintained. In another example implementation, when theacquisition rate of the second imaging modality is increased in step620, the acquisition of images from the first imaging modality may beterminated or suspended.

In one example embodiment, after having increased the acquisition rateof the second imaging modality based on the determined that an imagemeasure has met pre-selected criterion, steps 610 and 615 may beperformed to assess whether or not the image measure associated with thesecond imaging modality continues to meet the pre-selected criterion. Inthe event that the image measure fails to meet the pre-selectedcriterion, the acquisition rate of the second imaging modality may bereduced, and the method may be repeated (starting with step 605).

In one example embodiment, additional actions may be taken after havingdetermined that the image measure associated with the second imagingmodality satisfies the pre-selected criterion. For example, in a mannersimilar to the previously described embodiments, one or more devicesthat are used during the medical procedure may be reconfigured (e.g. byobtaining new configuration parameters from configuration dataassociating the configuration of one or more devices with the assessmentof the criterion in step 615). In one example implementation, a userinterface that displays the images obtained from the first imagingmodality may be re-configured to additionally or alternatively displaythe images from the second imaging modality.

The additional images from the second imaging modality may be displayedaccording to a wide variety of different configurations, such asdisplaying the images from the first and second imaging modalities in aside-by-side configuration or in an overlaid configuration (optionallyafter having registered the images from the first imaging modality withthose of the second imaging modality).

It is also noted that in some embodiments, a method based on that shownin FIG. 13A may be implemented based solely on the intermittentacquisition of images from one imaging modality. This embodiment isillustrated in FIG. 13B, which provides an adaptive and interoperativemethod of controlling the acquisition rate of images pertaining to animaging modality. In step 622, an imaging modality is employed tointraoperatively obtain images. The images are intermittently obtainedat an initial pre-selected frame rate. The one or more images areprocessed to obtain an image measure in step 624. In step 626, the imagemeasure is compared to pre-selected criterion, and if the criterion ismet, the acquisition rate of the imaging modality is increased.

Referring now to FIG. 14, a flow chart is shown illustrating an exampleimplementation of a method of intraoperatively and adaptivelycontrolling the acquisition of images from white light and hyperspectralimaging modalities. It will be understood that these imaging modalitiesare merely provided as examples. White light images are intraoperativelyobtained, and in step 632, one or more hyperspectral images areintermittently obtained while obtaining the white light images. Forexample, one or more hyperspectral images may be obtained at aprescribed initial acquisition rate, such as, for example, a ratebetween once per minute and once per second.

Hyperspectral images may be obtained via a separate hyperspectralimaging device. In one example implementation, a hyperspectral imagingdevice may share one or more components with the white light imagingdevice. For example, as illustrated in FIG. 5A, imaging optics assembly260 may be shared by both camera 255 (which, in the present example,would be employed for white light imaging), and by a hyperspectraldetector that interfaces with optical system 250 through auxiliaryimaging modality assembly 275.

The one or more hyperspectral images that are obtained in step 632 inFIG. 14 are then processed to obtain an image measure that may beemployed to determine whether or not to increase the acquisition rate ofhyperspectral images. The image measure can be associated with a widerange of metrics associated with the suitability or feasibility ofperforming hyperspectral imaging, including one or more of thosedescribed above.

In the present example, the image measure is obtained by processing thehyperspectral image data in order to identify the presence of one ormore spectral signatures, as shown in step 634. The processing of thehyperspectral images may be performed as described above in relation toFIG. 11A, in which one or more regions are identified having a spectralsimilarity among pixels. Briefly, per-pixel spectral data from thehyperspectral images may be processed to identify groups of adjacentpixels having a similar spectral response, thereby spectrally segmentingthe image into one or more regions of similar spectral content. Pixelswith similar spectral content may be identified based on the calculationof a spectral similarity measure between adjacent pixels. If a region isidentified as having spectral similarity within the pixels spanning theregion, then the average spectral response for the pixels within theregion may be employed as the spectral signature.

A spectral similarity measure is the evaluated between an identifiedspectral signature and one or more reference spectra, as shown in step636. For example, the spectral signature may be compared to one or morereference spectra, where the reference spectra are each associated witha given tissue type, fluid type, or a given chemical or biologicalcomposition. For example, the spectral signature may be compared withone or more pathological tissue types, such as different types oftumors. The comparison may be performed using a similarity measure asdescribed above. In some embodiments, the comparison may be performedonly for one or more regions having a number of pixels that exceeds aminimum value, such that similarity assessment is only performed forregions beyond a threshold size.

If the spectral similarity measure satisfies a pre-selected criterionfor one of the reference spectra (e.g. exceeds a pre-selected threshold)in step 640, then the acquisition rate of the second imaging modality isincreased as shown at step 642. On the other hand, if the pre-selectedcriterion is not met, then the process is repeated and additionalhyperspectral images are obtained in 632 and subsequently evaluated forspectral similarity with the reference spectra.

In another example implementation, the methods illustrated in FIGS. 13Aand 13B may be employed to adaptive control the use of fluorescenceimaging. For example, in one example implementation, the methodillustrated in FIG. 13A may be performed such that the first imagingmodality is white light imaging, and the second imaging modality isfluorescence imaging.

An example implementation of this method is shown in FIG. 15. In step652, one or more fluorescence images are intermittently obtained whileobtaining while light images. Fluorescence image acquisition may beinterleaved with white light image acquisition, in order to avoidcrosstalk between the two modalities. The fluorescence images areprocessed to calculate an image measure associated with the intensity ofthe fluorescence signal. For example, the image measure may be obtainedby calculating the net fluorescence intensity for all image pixels.

Alternatively, a spatially resolved measure of fluorescence intensitymay be calculated. For example, as shown in step 654, the per-pixelfluorescence intensity data may be processed to identify groups ofadjacent pixels having a fluorescence intensity exceeding a pre-selectedthreshold value, thereby segmenting the image into one or more regionshaving a fluorescence intensity above the threshold value. In oneexample, an identified region may be required to have a minimum numberof pixels, such only regions greater than a minimum area are considered.If one or more regions are identified in step 656, the rate ofacquisition of fluorescence images is increase.

In other example implementations involving fluorescence imaging, othermeasures associated with the fluorescence image may additionally oralternatively be obtained and compared to pre-selected criteria in orderto determine whether or not to increase the fluorescence imageacquisition rate. For example, a measure associated with thesignal-to-noise ratio of one or more fluorescence images may beobtained, and the fluorescence image acquisition rate may be increasedif the measure exceeds a pre-selected threshold. In another example, thefluorescence images may be spectrally resolved (e.g. using hyperspectralfluorescence detection) and the hyperspectral image processing methodspertaining to FIG. 14 may be employed.

The preceding embodiments involving the intermittent and intraoperativesampling of images from an imaging modality, and the processing of theimages in order to determine whether or not to increase the acquisitionrate of the images. In another example embodiment, images from a firstimaging modality may be obtained and processed in order to trigger theuse of a second imaging modality.

FIG. 16 provides a flow chart illustrating an example of such anembodiment. In step 705, a first imaging modality is employed tointraoperatively obtain images. The images are then processed in step710 in order to obtain an image measure associated with the feasibilityor suitability of performing imaging with a second imaging modality.Examples of such image measures are described below. The image measureis then compared to a pre-selected criterion in step 715, and if thecriterion is met, images are subsequently acquired with the secondimaging modality, as shown at step 720.

As in the preceding embodiment, the first imaging modality and thesecond imaging modality need not be associated with two separate imagingdevices, and in some example implementations, an imaging device that isinitially configured to obtain images using a first imaging modality maybe adaptively and dynamically configured to obtain images with a secondimaging modality by modifying the imaging device. For example, anoptical imaging device may be dynamically switched to fluorescence modevia the introduction of one or more filters into the optical beam path.In another example, an intraoperative magnetic resonance imaging systemmay be dynamically modified to switch between different modes ofoperation (e.g. T1 vs. T2 weighted images) via changes to the transmitand receive sequence.

As described above, the image measure may be associated with thefeasibility or suitability of imaging with the second imaging modality.The image measure can be obtained according to a wide range of methods.For example, in some example implementations, the image measure may beassociated with an impairment of the performance of the first imagingmodality, such that when the image measure exceeds a pre-selectedthreshold, it may be beneficial to switch to the second imagingmodality. For example, as described below, the first imaging modalitymay be an optical imaging modality that suffers a performancedegradation in the presence of glare, and the second imaging modalitymay be insensitive or less sensitive to glare. In such a case, when theimage measure has a value that is associated with glare, the criterionin step 715 will trigger the acquisition of images using the secondimaging modality.

In another example embodiment, an image measure may be associated withthe determination of context measure, as described in the precedingembodiments. For example, an image measure that is obtained may providean indication of the current phase of a surgical procedure, as describedabove. In such a case, the pre-selected criterion that is evaluated instep 715 may include a list of phases of the medical procedure for whichthe second imaging modality is desirable or suitable.

In another example embodiment, the image measure may be associated witha performance measure of the second imaging modality. For example, theimage measure may involve a determination of a measure ofsignal-to-noise ratio of imaging with the second imaging modality, suchthat when the imaging measure associated with the signal-to-noise ratioexceeds the pre-selected criterion in step 715, the acquisition rate ofthe second imaging modality is increased. Another example of aperformance-related image measure is a measure of the intensity of thesignal that is obtained with the second imaging modality. Yet anotherexample of a performance-related image measure is the amount of signalwithin a given frequency range or spectral range. These performancemeasures may be evaluated on a global basis using one or morestatistical measures, or may be evaluated on a local or regional basis.For example, the pre-selected criterion evaluated in step 640 in FIG. 14may require that a given performance threshold is satisfied by apre-selected fraction of the pixels forming the image obtained via thesecond imaging modality. It is further noted that a plurality of imagesmay be obtained from the second imaging modality, and an image measuremay be obtained by processing the plurality of images (for example, viaaveraging the images).

It will be understood that while the preceding paragraphs describe theuse of a single image measure, multiple image measures, and associatedcriterion, may be processed in order to determine whether or not toincrease the acquisition rate of the second imaging modality.

In one example implementation as shown in FIG. 16, when the acquisitionimages with the second imaging modality is initiated in step 720, theacquisition rate of the first imaging modality may be reduced. Inanother example implementation, when the acquisition images with thesecond imaging modality is initiated in step 720, the acquisition rateof the first imaging modality may be maintained. In another exampleimplementation, when the acquisition images with the second imagingmodality is initiated in step 720, the acquisition of images from thefirst imaging modality may be terminated or suspended.

In one example embodiment, after having initiated the acquisition of theimages from the second imaging modality based on the determined that animage measure has met pre-selected criterion, steps 710 and 715 may beperformed to assess whether or not the image measure associated with thesecond imaging modality continues to meet the pre-selected criterion. Inthe event that the image measure fails to meet the pre-selectedcriterion, the acquisition of images with the second imaging modalitymay be reduced or terminated, and the method may be repeated (startingwith step 705).

In one example embodiment, additional actions may be taken after havingdetermined that the image measure associated with the second imagingmodality satisfies the pre-selected criterion. For example, in a mannersimilar to the previously described embodiments, one or more devicesthat are used during the medical procedure may be reconfigured (e.g. byobtaining new configuration parameters from configuration dataassociating the configuration of one or more devices with the assessmentof the criterion in step 615 as shown in FIG. 13A). In one exampleimplementation, a user interface that displays the images obtained fromthe first imaging modality may be re-configured to additionally oralternatively display the images from the second imaging modality. Theadditional images from the second imaging modality may be displayedaccording to a wide variety of different configurations, such asdisplaying the images from the first and second imaging modalities in aside-by-side configuration or in an overlaid configuration (optionallyafter having registered the images from the first imaging modality withthose of the second imaging modality).

FIG. 17 illustrates an example implementation of the method outlined inFIG. 16, in which the first imaging modality employs white lightimaging, and the second imaging modality employs cross-polarizedimaging. In this example implementation, the system automaticallyswitches to cross-polarized imaging device when a pre-selected criterionassociated with the detection of glare in the images from the whitelight imaging modality. As noted above, a significant issue with currentsurgical optical systems and devices is glare caused by fluids thatreflect illumination within a surgical cavity. The glare can causeimbalance in the dynamic range of an imaging camera, causing the upperrange of the camera's dynamic range to become saturated. In addition,glare can cause the illumination intensity across the frequency spectrumof an imager to be unbalanced, depending on the illumination andconditions of the medical procedure.

Accordingly, in the example method illustrated in FIG. 17, white lightimages of a region of interest are initially obtained during a medicalprocedure, as shown at step 732. For example, such images may beobtained using camera 255 of optical system 250 shown in FIG. 5A. Instep 734, the white light images are processed to calculate an imagemeasure associated with the presence of glare in the white light images.For example, this may be performed by identifying one or more of regionswithin the image (groups of adjacent pixels) having an intensity valueabove a pre-selected intensity, where the pre-selected intensity isindicative of glare conditions. In one or more of such regions areidentified, in step 736 (optionally where any given region has an areaexceeding a pre-selected minimal area), then images are subsequentlyacquired using a cross-polarization imaging modality, as shown at step738. The cross-polarization images may be intermittently obtained whilecontinuing to obtaining the white light images.

Cross-polarized images may be obtained via a separate cross-polarizationimaging device, or may be obtained by modifying the optical device thatis employed for white light imaging. For example, the device employedfor white light imaging may be modified by intraoperatively inserting,into the beam path of an illumination device, a first polarizer, andintroducing, into the beam path of the optical imaging device, a secondpolarizer (an analyzer), where the first and second polarizers areoriented in a crossed configuration for performingpolarization-sensitive imaging. In some example implementations,cross-polarization imaging may be performed using a high frequencypolarization state actuation and deactivation device, a beam splitterand an alternate camera, or a beam splitter with same camera. In anexample implementation in which a second imaging device is obtained forperforming cross-polarized imaging, one or more cross-polarized imagesmay be obtained concurrently with the acquisition of white light images.

The additional images from the cross-polarization imaging modality maybe displayed according to a wide variety of different configurations,such as displaying the images from the white light andcross-polarization imaging modalities in a side-by-side configuration,in an overlaid configuration, or in a configuration in which thehigh-glare regions identified in the white light images are replacedwith image data obtained from cross-polarization imaging.

FIG. 18 illustrates another example implementation of the methodoutlined in FIG. 16, in which the first imaging modality employshyperspectral imaging, and the second imaging modality employsnear-infrared imaging. In this example implementation, the systemautomatically switches to near-infrared imaging device when apre-selected criterion associated with the detection of a spectralsignature in the hyperspectral images is satisfied.

In step 742, one or more hyperspectral images are intraoperativelyobtained. The one or more hyperspectral images are the processed, instep 744, in order to identify one or more spatial regions having asimilar spectral signature. Example methods for identifying suchregions, and a characteristic spectral signature for a given region, aredescribed in detail above.

In step 746, the spectra signature from each identified region iscompared to one or more reference spectra, where the reference spectrapertain to tissue types, fluids, material or biological compositionsthat are known to be suitable or feasible for near-infrared imaging.

In another example, the spectral signature may be processed to providean image measure associated with the relative spectral intensity withinone or more spectral bands, where the spectral bands are known to beassociated with materials that do not absorb near-infrared light. Inother words, the spectral signature may be processed to identify,directly or indirectly, the presence of a material that would supportdeeper image penetration via near-infrared imaging. The image measuremay be compared to pre-selected criterion in order to selectivelytrigger the use of the near-infrared imaging modality.

In one example implementation, multiple image measures may be obtainedand employed. For example, image measures associated with the presenceof both near-infrared absorbing substances and near-infrared transparentsubstances may be combined to determine whether or not to trigger theuse of the near-infrared imaging modality.

The spectral similarity can be determined, for example, based on thecalculation of a spectral similarity measure, as described in detailabove. In the event that sufficient spectral similarity is found tooccur between a spectral signature from the hyperspectral images and thereference spectra, then the acquisition of near-infrared images istriggered in step 748.

It will be understood that the preceding example involving the analysisof images from one imaging modality to trigger the acquisition of imagesfrom another imaging modality are provided as non-limiting heuristicexamples, and that the method may be adapted to various combinations ofimaging modalities without departing from the scope of the presentdisclosure.

The additional images from the near-infrared imaging modality may bedisplayed according to a wide variety of different configurations, suchas displaying the images from the hyperspectral and near-infraredimaging modalities in a side-by-side configuration, in an overlaidconfiguration, or in a configuration in which the regions identified inthe hyperspectral images are replaced with image data obtained fromnear-infrared imaging.

In one embodiment, the adaptive system may be utilized to configure theimaging device (video scope) to reduce glare conditions at the surgicalsite of interest. The process, as illustrated in FIG. 22. The first step(2200) in this process is to acquire a visible light image from theimaging device (for example an external scope). The following step(2205) in the process is to scan the signal from each pixel within theregion of interest on the acquired image, assigning each pixel anintensity value based on the dynamic range of the imaging device (in aport based surgery this would be the distal end of the port, where thesurgical site is located and tumor resection is being performed by thesurgeon). Using these values, the third stage in the process (2210) isto create a matrix using these pixels, where each element of the matrixcorresponds to a pixel location on the image and the correspondingmatrix element locations are conserved with respect to their pixelcounterparts.

The next step in the process (2215) is to identify areas (defined by >xnumber of pixels in a row in the X direction and >y number of pixels ina row in the Y direction (X and Y being chosen values for a minimumarea)) of groups of pixels with similar intensities. Step (2220) is toassign each area a value from 1 to n, continuing with the flow chart.

In step (2225), n is to zero, so that in the following step (2230), then=1 case is considered (since step 2230 involves assessing area numbern+1). The following steps (2230, 2235, 2240, and 2245) pertain to a loopthat determines if each identified area (1 to n) intensity level isindicative of glare conditions. It should be noted that the glarecondition can be chosen by a user and input into the adaptive system orpredetermined by the adaptive system and is defined by an intensitythreshold. The loop stores each area which is indicative glareconditions (i.e. has an intensity above the given threshold) in anarray. The next step before continuing (2250) is to check if there areany areas with glare conditions if not the process returns to the firststep (2200) and is repeated. If there are glare conditions, the nextstep (2255) indicates that polarized imaging should begin gettingacquired by the imaging device (for example the external scope). In thenext step (2260) the imaging stream acquired from the visible lightimaging device is segmented according to the areas as defined by thearray and located on the matrix. The final step (2265) is to overlaythose identified areas using the polarization imaging stream acquiredusing the imaging device. This overlay effectively reduces the glareconditions for the surgeon as they perform the surgery.

In an embodiment the FIG. 24 illustrates another example method forperforming adaptive system may be utilized to configure the imagingdevice (video scope) to reduce glare conditions. Unlike theintensity-based example method shown in FIG. 22, the present examplemethod employs spectral analysis for the detection of conditionsassociated with blood occlusion in the surgical site of interest. Theprocess in which this is achieved is depicted in FIG. 24. The first step(2400) in this process is to acquire a visible light image from theimaging device (for example an external scope). The following step(2405) in the process is to scan each pixel within the region ofinterest on the acquired image assigning each pixel a wavelength (color)spectrum value based on the appropriate range of the visible lightspectrum. Using these values the third stage in the process (2410) is tocreate a matrix using these pixels, where each element of the matrixcorresponds to a pixel location on the image and the correspondingMatrix element locations are conserved with respect to their pixelcounterparts. The next step in the process (2415) is to identify areas(defined by >x number of pixels in a row in the X direction and >ynumber of pixels in a row in the Y direction (X and Y being chosenvalues for a minimum area)) of groups of pixels with similar wavelengthspectrums. Step (2420) is to assign each area a value from 1 to n,continuing with the flow chart.

In step (2425), n is set initially to zero, so that in the followingstep (2230), the n=1 case is considered (since step 2430 involvesassessing area number n+1). The following steps (2430, 2435, 2440, and2445) pertain to a loop that determines if each identified area (1 to n)intensity level is indicative of blood occlusion. It should be notedthat the blood occlusion can be identified by comparing the assignedwavelength spectrum values to a known value for blood (corresponding toits color). The loop stores each area which is indicative bloodocclusion (i.e. has the same wavelength spectrum as blood) in an array.Before continuing the following step (2250) is to check if there are anyareas with blood occlusion if not the process returns to the first step(2400) and is repeated. If there is blood occlusion, the next step(2455) indicates that whether the image acquisition based onnear-infrared (NIR) imaging should commence (for example by the imagingdevice acquiring NIR images with the external scope). In the next step(2460) the imaging stream acquired from the visible light imaging deviceis segmented according to the areas as defined by the array and locatedon the matrix. The final step (2465) is to overlay those identifiedareas using the NIR imaging stream acquired using the imaging device.This overlay effectively increases the ability of the surgeon to seethrough the blood as they perform the surgery.

FIG. 23 is a flow chart depicting the actuation of an example method ofutilizing polarization sensitive imaging to determine surfacestructures. One particular use of this type of imaging would be its useto decipher surface structures that would be representative of vitalregions within a patient's brain such as fiber tracts or within thepatient's body such as tendons. The first step in this process (2300) isto acquire the spatial position of the instrument (such as a resectiondevice) in the spatially registered intraoperative reference frameassociated with the Navigation system (i.e. using the tracking devicewithin used by the Navigation system). The second step (2310) is tospatially register the position of the preoperative 3D MRI image data inthe common coordinate frame from the Navigation system.

The following two steps (2320) and (2330) are used to determine whetherthe instrument comes close to a fiber tract, if the instrumentapproaches a fiber tract in close proximity (e.g. within a pre-selecteddistance). If it is deemed that the instrument is not close to a fibertract, the process returns to the initial step (2300) and repeats. If itis determined that the instrument comes close to a fiber tract, thesystem control and processing unit 400 configures the imaging device tobegin acquiring polarization sensitive imaging (2350) and displays theimaging to the surgeon (2340). This allows the surgeon to potentiallydecipher any brain tracts that he may damage while performing resectionand allows him to stay clear of those vital areas. The specificembodiments described above have been shown by way of example, and itshould be understood that these embodiments may be susceptible tovarious modifications and alternative forms. It should be furtherunderstood that the claims are not intended to be limited to theparticular forms disclosed, but rather to cover all modifications,equivalents, and alternatives falling within the spirit and scope ofthis disclosure.

Therefore what is claimed is:
 1. A computer implemented method ofadaptively and intraoperatively configuring a device used during amedical procedure, the method comprising: identifying a medicalinstrument during the medical procedure; obtaining one or morecustomized configuration parameters for adaptively configuring thedevice during the medical procedure, where the customized configurationparameters are selected based on the identity of the medical instrument;and configuring the device according to the customized configurationparameters.
 2. The method according to claim 1 wherein the medicalinstrument is automatically identified.
 3. The method according to claim2 wherein the medical instrument is identified by: detecting a signalfrom one or more fiducial markers associated with the medicalinstrument; and processing the signal to identify the medicalinstrument.
 4. The method according to claim 3 wherein the one or morefiducial markers are selected from the group consisting of passivemarkers, active markers, glyphs and RFID tags.
 5. The method accordingto claim 3 wherein the medical instrument is identified based on imageanalysis employing a known shape of the medical instrument.
 6. Themethod according to claim 1 wherein identifying one or more medicaldevices comprises receiving input from an operator, the inputidentifying the medical instrument.
 7. The method according to any oneof claims 1 to 6 wherein identifying the medical instrument comprisesidentifying the type of the medical instrument.
 8. The method accordingto any one of claims 1 to 6 wherein identifying the medical instrumentcomprises uniquely identifying the medical instrument.
 9. The methodaccording to claim 8 wherein uniquely identified medical instrument isassociated with one or more operators.
 10. The method according to claim8 wherein uniquely identified medical instrument associated isassociated with one or more regions of use.
 11. The method according toany one of claims 1 to 10 further comprising: receiving inputidentifying one or more operators present during the medical procedure;wherein the customized configuration parameters are further associatedwith the identity of the one or more operators.
 12. The method accordingto any one of claims 1 to 10 further comprising: receiving inputidentifying the medical procedure; wherein the customized configurationparameters are further associated with the medical procedure.
 13. Themethod according to any one of claims 1 to 12 further comprising:determining a current stage of the medical procedure; wherein thecustomized configuration parameters are further associated with thecurrent stage of the medical procedure.
 14. The method according to anyone of claims 1 to 13 wherein the customized configuration parametersare obtained from a data storage device comprising customizedconfiguration parameters for adaptively configuring a plurality ofdevices during the medical procedure, the method further comprising:identifying one or more additional medical instruments employedconcurrently during the medical procedure; wherein the one or morecustomized configuration parameters are selected based on the identityof identified medical instruments.
 15. The method according to claim 14wherein the customized configuration parameters stored in the datastorage device are ranked based on the identity of different medicalinstruments, and wherein the customized configuration parameters areobtained for the identified instrument having the highest ranking. 16.The method according to any one of claims 1 to 15 wherein the device isan auxiliary device.
 17. The method according to any one of claims 1 to16 wherein the device is an illumination device.
 18. The methodaccording to any one of claims 1 to 16 wherein the device is computerhardware for generating a user interface on a display.
 19. The methodaccording to any one of claims 1 to 16 wherein the device is a roboticpositioning device.
 20. The method according to any one of claims 1 to16 wherein the device is an imaging device.
 21. The method according toclaim 20 wherein the customized configuration parameters comprise one ormore of colour balance, brightness, depth of field, magnification, fieldof view, working distance, and illumination conditions.
 22. The methodaccording to claim 20 wherein the imaging device is a multimodal imagingdevice, and wherein the customized configuration parameters comprise aselection of an imaging modality.
 23. The method according to any one ofclaims 1 to 22 wherein the medical procedure is a surgical procedure.24. A system for adaptively and intraoperatively configuring a deviceused during a medical procedure, comprising: a data storage devicecomprising customized configuration parameters for adaptivelyconfiguring one or more devices during the medical procedure; a controland processing system interfaced with the device and the data storagedevice, said control and processing system comprising one or moreprocessors and memory coupled to said one or more processors, saidmemory storing instructions, which, when executed by said one or moreprocessors, causes said one or more processors to perform operationscomprising: identifying a medical instrument during the medicalprocedure; obtaining, from the data storage device, one or morecustomized configuration parameters for adaptively configuring thedevice during the medical procedure, where the customized configurationparameters are customized based on the identity of the medicalinstrument; and configuring the device according to the customizedconfiguration parameters.
 25. A computer implemented method ofadaptively configuring a device used during a medical procedure, themethod comprising: obtaining one or more images of a region of interestassociated with the medical procedure; processing the one or more imagesto identify a context measure associated with the current state of themedical procedure; obtaining one or more customized configurationparameters for adaptively configuring the device during the medicalprocedure, where the customized configuration parameters are customizedbased on the context measure; and configuring the device according tothe customized configuration parameters.
 26. The method according toclaim 25 wherein the context measure is associated with a current phaseof the medical procedure.
 27. The method according to claim 25 whereinthe context measure is associated with the type of medical procedure.28. The method according to claim 25 wherein the context measure isassociated with the presence of one or more tissue types within theregion of interest.
 29. The method according to claim 25 wherein thecontext measure is associated with the presence of one or more fluidswithin the region of interest.
 30. The method according to claim 25wherein one of the one or more fluids is a biological fluid.
 31. Themethod according to claim 25 wherein the biological fluid is blood. 32.The method according to claim 25 wherein the context measure isassociated with the presence of one or more medical instruments withinthe region of interest.
 33. The method according to claim 26 whereinprocessing the one or more images to identify the context measureassociated with the current phase of the medical procedure comprises:acquiring hyperspectral image data from a region of interest; processinghyperspectral image data to identify one or more regions with similarspectral response among pixels; comparing an average spectral responsefrom each region to reference spectra associated with different phasesof the medical procedure; and identifying the phase of medical procedurebased on similarity between average spectral response and referencespectra.
 34. The method according to claim 26 wherein processing the oneor more images to identify the current phase of the medical procedurecomprises: processing the one or more images to identify one or moretissue types present within the region of interest; and determining thephase of the medical procedure based on the one or more tissue types.35. The method according to claim 26 wherein processing the one or moreimages to identify the current phase of the medical procedure comprises:performing image analysis to identify one or more medical instrumentsbased on the known shape of one or more medical instrument; anddetermining the phase of the medical procedure based on the presence ofthe one or more medical instruments.
 36. The method according to claim25 wherein the one or more images are optical images.
 37. The methodaccording to claim 36 wherein at least one of the optical images arebased on detection in at least the visible spectrum.
 38. The methodaccording to claim 36 wherein at least one of the optical images arebased on detection in at least the infrared spectrum.
 39. The methodaccording to claim 25 wherein at least one of the images is an opticalcoherence tomography image.
 40. The method according to claim 25 whereinthe images are obtained, at least in part, by magnetic resonanceimaging.
 41. The method according to claim 25 wherein the images areobtained, at least in part, by ultrasound imaging.
 42. The methodaccording to any one of claims 25 to 41 wherein processing the imagescomprises processing two or more images from different imagingmodalities.
 43. The method according to any one of claims 25 to 42wherein the device is a surgical tool.
 44. The method according to anyone of claims 25 to 42 wherein the device is an auxiliary device. 45.The method according to any one of claims 25 to 42 wherein the device isan illumination device.
 46. The method according to according to any oneof claims 25 to 42 wherein the device comprises computer hardware forgenerating a user interface on a display.
 47. The method according toaccording to any one of claims 25 to 42 wherein the device comprises arobotic positioning device.
 48. The method according to according to anyone of claims 25 to 42 wherein the device is an imaging device.
 49. Themethod according to any one of claims 25 to 48 wherein controlling thedevice comprises: obtaining one or more customized configurationparameters for adaptively configuring the device during the medicalprocedure, where the customized configuration parameters are customizedbased on the current phase of the medical procedure; and configuring thedevice according to the customized configuration parameters.
 50. Themethod according to claim 49 wherein the device is an imaging device,and wherein the customized configuration parameters comprise one or moreof colour balance, brightness, depth of field, magnification, field ofview, working distance, and illumination conditions.
 51. The methodaccording to claim 49 wherein the imaging device is a multimodal imagingdevice, and wherein the customized configuration parameters comprise aselection of an imaging modality.
 52. The method according to any one ofclaims 25 to 51 wherein the medical procedure is a surgical procedure.53. A system for adaptively and intraoperatively configuring a deviceused during a medical procedure, comprising: a data storage devicecomprising customized configuration parameters for adaptivelyconfiguring one or more devices during the medical procedure; a controland processing system interfaced with the device and the data storagedevice, said control and processing system comprising one or moreprocessors and memory coupled to said one or more processors, saidmemory storing instructions, which, when executed by said one or moreprocessors, causes said one or more processors to perform operationscomprising: obtaining one or more images of a region of interestassociated with the medical procedure; processing the one or more imagesto identify a context measure associated with the current state of themedical procedure; obtaining one or more customized configurationparameters for adaptively configuring the device during the medicalprocedure, where the customized configuration parameters are customizedbased on the context measure; and configuring the device according tothe customized configuration parameters.
 54. A computer implementedmethod of adaptively controlling a first imaging modality and a secondimaging modality during a medical procedure, the method comprising:while obtaining first images with the first imaging modality,intermittently obtaining one or more second images with the secondimaging modality; processing the second images to calculate, for aplurality of regions within the second images, an image measureassociated with the second imaging modality; and in the event that theimage measure for one or more regions is within a pre-selected range,increasing the rate of acquisition of the second images.
 55. The methodaccording to claim 54 further comprising, in the event that the imagemeasure for one or more regions is within a pre-selected range,continuously acquiring the second images.
 56. The method according toclaim 54 further comprising, in the event that the image measure for oneor more regions is within a pre-selected range, reducing the rate ofacquisition of the first images.
 57. The method according to claim 54further comprising, in the event that the image measure for one or moreregions is within a pre-selected range, terminating acquisition of thefirst images.
 58. The method according to any one of claims 54 to 57wherein the second images are intermittently obtained at a pre-selectedrate.
 59. The method according to any one of claims 54 to 57 wherein anillumination source associated with the first imaging modality is turnedoff while obtaining the second images with the second imaging modality.60. The method according to any one of claims 54 to 59 wherein the imagemeasure is associated with the presence of one or more tissue typesidentifiable with the second imaging modality.
 61. The method accordingto claim 60 wherein the tissue type comprise one of tumor tissue andwhite matter.
 62. The method according to any one of claims 54 to 61wherein the second imaging modality comprises one of fluorescenceimaging, Raman imaging, hyperspectral imaging, optical coherencetomography imaging, and polarization-sensitive optical imaging.
 63. Themethod according to any one of claims 54 to 61 wherein the secondimaging modality provides hyperspectral images, wherein the method ofprocessing the second images comprises: processing the second images tocalculate, for a plurality of regions within the second images, an imagemeasure associated with the presence of a pre-selected spectralsignature.
 64. The method according to any one of claims 54 to 63wherein the first imaging modality employs optical imaging in thevisible spectrum.
 65. The method according to any one of claims 54 to 64further comprising overlaying the portion of the second imagescorresponding to the one or more regions with the first images.
 66. Themethod according to any one of claims 54 to 64 further comprisingdisplaying both the first images and the second images.
 67. A system foradaptively controlling one or more imaging devices during a medicalprocedure, comprising: a control and processing system interfaced withthe one or more imaging devices, said control and processing systemcomprising one or more processors and memory coupled to said one or moreprocessors, said memory storing instructions, which, when executed bysaid one or more processors, causes said one or more processors toperform operations comprising: obtaining first images with a firstimaging modality and intermittently obtaining one or more second imageswith a second imaging modality; processing the second images tocalculate, for a plurality of regions within the second images, an imagemeasure associated with the second imaging modality; and in the eventthat the image measure for one or more regions is within a pre-selectedrange, increasing the rate of acquisition of the second images.
 68. Acomputer implemented method of adaptively controlling one or moreimaging devices during a medical procedure, the method comprising: a)obtaining one or more first images with a first imaging modality; b)processing the first images to calculate, for a plurality of regionswithin the first images, an image measure associated with thesuitability of a second imaging modality; c) in the event that the imagemeasure for one or more regions lies within a pre-selected range,acquiring one or more second images with the second imaging modality.69. The method according to claim 68 wherein the first imaging modalityemploys non-polarized optical imaging, and the second imaging modalityemploys cross-polarized optical imaging, and wherein the image measureis associated with glare.
 70. The method according to claim 69 whereinthe image measure is calculated by determining the relative intensity ofthe regions, and wherein the pre-selected range corresponds to intensityvalues indicative of glare.
 71. The method according to claim 68 whereinthe image measure is associated with a spectral signature known to besuitable for imaging with the second imaging modality.
 72. The methodaccording to claim 71 wherein the first imaging modality employs in thevisible spectrum, and the second imaging modality employs near infraredimaging, and wherein the image measure is associated with a spectralsignature known to be suitable for near infrared imaging.
 73. The methodaccording to claim 72 further wherein the spectral signature isassociated with a substance or tissue type.
 74. The method according toclaim 72 further wherein the spectral signature is associated with asubstance or tissue type or substance that is transparent to nearinfrared imaging.
 75. The method according to claim 74 further whereinthe substance is blood.
 76. The method according to claim 72 furtherwherein the first imaging modality employs hyperspectral imaging. 77.The method according to claim 72 further wherein the first imagingmodality employs colour imaging, and wherein the spectral signaturedetermined based on the relative intensity of different image colours.78. The method according to any one of claims 72 to 77 wherein the imagemeasure is calculated by determining the relative image intensity withina pre-selected spectral region.
 79. The method according to any one ofclaims 68 to 78 further comprising repeating steps a) to c) one or moretimes.
 80. The method according to any one of claims 68 to 78 whereinstep c) further comprises continuously acquiring additional secondimages with the second imaging modality.
 81. The method according to anyone of claims 68 to 80 further comprising overlaying the portion of thesecond images corresponding to the one or more regions with the firstimages.
 82. The method according to any one of claims 68 to 80 furthercomprising displaying both the first images and the second images.
 83. Asystem for adaptively controlling a one or more imaging devices during amedical procedure, comprising: a control and processing systeminterfaced with the one or more imaging devices, said control andprocessing system comprising one or more processors and memory coupledto said one or more processors, said memory storing instructions, which,when executed by said one or more processors, causes said one or moreprocessors to perform operations comprising: obtaining one or more firstimages with a first imaging modality; processing the first images tocalculate, for a plurality of regions within the first images, an imagemeasure associated with the suitability of a second imaging modality; inthe event that the image measure for one or more regions lies within apre-selected range, acquiring one or more second images with the secondimaging modality.
 84. A method of performing adaptive illumination whileperforming optical imaging during a medical procedure, the methodcomprising: determining the field of view of an optical imaging deviceemployed during the medical procedure; determining configurationparameters of an illumination source for improving the homogeneity ofillumination within the field of view; configuring the illuminationsource according to the configuration parameters.
 85. The methodaccording to claim 84 wherein the configuration parameters aredetermined by: a) obtaining an image with the imaging device; b)processing the image to determine an image measure associated with thehomogeneity of illumination; c) varying the position and/or orientationof the illumination source; and d) repeating steps (a)-(c) until theimage measure lies within a pre-selected range.
 86. The method accordingto claim 84 wherein the configuration parameters are determined by: a)determining the current position and orientation of the illuminationsource relative to the field of view; b) calculating, based on a modelof the illumination source, an image measure associated with homogeneityof illumination within the field of view; c) varying the position and/ororientation of the illumination source within the model; and d)repeating steps b)-c) until the image measure lies within a pre-selectedrange.
 87. The method according to claim 86 wherein the model of theillumination source accounts for reflections or scattering from one ormore surfaces into the field of view.
 88. The method according to claim86 wherein the model of the illumination source is a non-sequential raytracing model.
 89. A system for performing adaptive illumination whileperforming optical imaging during a medical procedure, comprising: acontrol and processing system interfaced with the optical imaging deviceand the illumination source, said control and processing systemcomprising one or more processors and memory coupled to said one or moreprocessors, said memory storing instructions, which, when executed bysaid one or more processors, causes said one or more processors toperform operations comprising: determining the field of view of anoptical imaging device employed during the medical procedure;determining configuration parameters of an illumination source forimproving the homogeneity of illumination within the field of view;configuring the illumination source according to the configurationparameters.